Repository: openshift/runbooks
Pull request #243: add runbook for StorageAutoScaler alerts
Author: login: ShravaniVangur
Labels: name: approved
description: Indicates a PR has been approved by an approver from all required OWNERS files.
name: lgtm
description: Indicates that a PR is ready to be merged.
Created: 2025-03-18T08:03:33Z | Merged: 2025-03-27T10:03:23Z

Adds runbook help for StorageAutoScalingCapacityReached and StorageAutoScalingFailed alerts.


Diff:
diff --git a/alerts/openshift-container-storage-operator/StorageAutoScalingCapacityReached.md b/alerts/openshift-container-storage-operator/StorageAutoScalingCapacityReached.md
new file mode 100644
index 00000000..91c35bb3
--- /dev/null
+++ b/alerts/openshift-container-storage-operator/StorageAutoScalingCapacityReached.md
@@ -0,0 +1,45 @@
+# StorageAutoScalingCapacityReached
+
+## Meaning
+
+The Storage Capacity Limit is the maximum storage that can be provisioned for
+the system. `StorageAutoScalingCapacityReached` alert is fired when the complete
+storage space has been utilized and hence additional storage cannot be allocated
+for the device class in the particular storage cluster.
+
+## Impact
+
+The storage cannot be auto-scaled at the moment.
+
+## Diagnosis
+
+Retrieve status of `storageCapacityLimitReached` from the StorageAutoScaler CR.
+The `resource_name` and `resource_namespace` would be specified in the alert itself.
+
+```bash
+oc get storageautoscaler <resource_name> -n <resource_namespace> -o json | jq -r '.status.storageCapacityLimitReached'
+```
+
+If status of `storageCapacityLimitReached` is `true` then maximum capacity of
+the storage has been utilized.
+
+## Mitigation
+
+To address the issues mentioned above, you can apply one of the following solutions
+to scale the storage:
+
+- Increase storage capacity: Run the command below with the updated `storageCapacityLimit`:
+  ```bash
+  oc patch storageautoscaler <resource_name> -n <resource_namespace> --type=merge --patch '{"spec":{"storageCapacityLimit":<storageCapacityLimit>}}'
+  ```
+  Note: The `status.error.message` in the StorageAutoScaler CR will show the
+  minimum required increase for `storageCapacityLimit`.Ensure that the specified
+  capacity is at least that value or higher.
+
+- If auto scaling is no longer desired, disable the feature. Additional storage
+can be added when needed via the ODF console.To disable auto scaling:
+  ```bash
+  oc delete storageautoscaler <resource_name> -n <resource_namespace>
+  ```
+  
+
diff --git a/alerts/openshift-container-storage-operator/StorageAutoScalingFailed.md b/alerts/openshift-container-storage-operator/StorageAutoScalingFailed.md
new file mode 100644
index 00000000..f4834a6e
--- /dev/null
+++ b/alerts/openshift-container-storage-operator/StorageAutoScalingFailed.md
@@ -0,0 +1,79 @@
+# StorageAutoScalingFailed
+
+## Meaning
+
+During automatic storage scaling, failures may occur due to various factors. In
+such instances, the `StorageAutoScalingFailed` alert is fired.
+
+## Impact
+
+The storage cannot be scaled at the moment.
+
+## Diagnosis
+
+There are multiple stages where the storage configurations might not have matched
+the requested storage parameters.
+
+1. OSD pod is in `Pending` state:
+    If it is in `Pending` state with the following logs, there is a high chance
+    it is due to the exhaustion of the resources. Describe the pod in `Pending`
+    state and check for the events.
+    ```bash
+    oc describe pod <osd_pod> -n <namespace>
+    ```
+
+    If the output is similar to the example below, it indicates that resources
+    are exhausted:
+    ```bash
+    Warning  FailedScheduling  3m53s  default-scheduler  0/6 nodes are available: 1 Insufficient cpu, 2 node(s) didn't match Pod's node affinity/selector, 3 node(s) had untolerated taint {node-role.kubernetes.io/master: }. preemption: 0/6 nodes are available: 1 Insufficient cpu, 5 Preemption is not helpful for scheduling.
+    ```
+
+    If there are no errors, retrieve the pvc of the pod in `Pending` state.
+    ```bash
+    pvcName=$(oc get pod <osd_pod> -n <namespace> -o jsonpath='{.spec.volumes[*].persistentVolumeClaim.claimName}')
+    oc get pvc $pvcName -n <namespace> -o yaml
+    ```
+    Check for the health and events of the PVC for detailed information.
+
+2. PVC is in `Pending` state:
+
+    ```bash
+    oc get pvc <pvc_name> -n <namespace> -o yaml
+    ```
+
+    Inspect the events of the pvc. Additionally, retrieve the pv of the pvc if
+    it is still in `pending` state.
+    ```bash
+    oc get pvc <pvc_name> -n <namespace> -o jsonpath='{.spec.volumeName}'
+    ```
+
+3. PV is in `Pending` state:
+    ```bash
+    oc get pv <pv_name> -n <namespace> -o yaml
+    ```
+    If the pv is in `Pending` state, there might be a storage issue. Check the
+    events of the pv for more information.
+
+## Mitigation
+
+1. If resources are depleted and pods start failing, allocate additional
+resources to the storage cluster. Possible solutions include adding more nodes
+or scaling up CPU and memory on existing nodes.
+- For instructions on adding nodes, please refer to this document: [Adding a node](https://docs.redhat.com/en/documentation/red_hat_openshift_data_foundation/latest/html-single/scaling_storage/index#adding_a_node)
+
+2. Check upon the Ceph Storage: In case the OSDs are `down` or `out`, restart them
+and try again.
+
+3. During vertical scaling, if the pv takes time to resize, then the osds would not
+resize correctly. In this case, the osd pods can be restarted.
+
+4. Additionally, one could try to restart the rook operator if it is stuck reconciling:
+    ```bash
+    oc rollout restart deployment rook-ceph-operator -n <namespace>
+    ```
+
+    After restarting, look for errors or warnings in the logs of rook-ceph-operator:
+    ```bash
+    oc logs -l app=rook-ceph-operator -n <namespace>
+    ```
+    Monitor the status of the pods and cephcluster post restart.
\ No newline at end of file


Reviews:
user: login: parth-gr
body: For failure alert try to segregate into steps and its mitigation,

1) PVC issue

2) pv issue 

3) osd pod issue

4) resource issue 


state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-19T14:51:00Z
user: login: parth-gr
state: CHANGES_REQUESTED
author_association: MEMBER
submitted_at: 2025-03-21T19:08:51Z
user: login: ShravaniVangur
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-24T06:35:00Z
user: login: ShravaniVangur
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-24T06:48:10Z
user: login: parth-gr
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-24T06:58:14Z
user: login: parth-gr
state: CHANGES_REQUESTED
author_association: MEMBER
submitted_at: 2025-03-24T07:11:16Z
user: login: parth-gr
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-24T12:17:42Z
user: login: parth-gr
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-24T12:19:48Z
user: login: Madhu-1
state: COMMENTED
author_association: NONE
submitted_at: 2025-03-24T13:00:55Z
user: login: parth-gr
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-24T13:10:27Z
user: login: travisn
state: CHANGES_REQUESTED
author_association: NONE
submitted_at: 2025-03-24T18:10:50Z
user: login: parth-gr
state: APPROVED
author_association: MEMBER
submitted_at: 2025-03-25T13:58:02Z
user: login: parth-gr
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-25T13:58:52Z
user: login: travisn
state: CHANGES_REQUESTED
author_association: NONE
submitted_at: 2025-03-25T20:02:21Z
user: login: ShravaniVangur
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-26T05:23:32Z
user: login: parth-gr
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-26T06:11:12Z
user: login: parth-gr
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-26T06:12:26Z
user: login: parth-gr
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-26T06:13:06Z
user: login: parth-gr
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-26T06:17:31Z
user: login: parth-gr
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-26T06:18:30Z
user: login: parth-gr
state: CHANGES_REQUESTED
author_association: MEMBER
submitted_at: 2025-03-26T07:55:26Z
user: login: ShravaniVangur
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-26T09:13:17Z
user: login: parth-gr
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-26T11:38:54Z
user: login: travisn
state: COMMENTED
author_association: NONE
submitted_at: 2025-03-26T14:07:32Z
user: login: parth-gr
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-26T14:29:42Z
user: login: parth-gr
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-26T14:30:16Z
user: login: travisn
state: COMMENTED
author_association: NONE
submitted_at: 2025-03-26T17:26:12Z
user: login: ShravaniVangur
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-26T17:54:10Z
user: login: travisn
state: APPROVED
author_association: NONE
submitted_at: 2025-03-26T17:55:52Z
user: login: parth-gr
state: APPROVED
author_association: MEMBER
submitted_at: 2025-03-27T06:34:48Z
user: login: agarwal-mudit
state: APPROVED
author_association: NONE
submitted_at: 2025-03-27T09:58:34Z

Review comments:
diff_hunk: @@ -0,0 +1,28 @@
+# StorageAutoScalingCapacityReached
+
+## Meaning
+
+The Storage Capacity Limit is the maximum storage that can be provisioned for the system. `StorageAutoScalingCapacityReached` alert is fired when the complete storage space has been utilized and hence additional storage cannot be allocated for the device class in the particular storage cluster.
+
+## Impact
+
+The storage cannot be scaled at the moment.
+
+## Diagnosis
+
+Retrieve the StorageAutoScaler Custom Resource for which the Storage Capacity is trying to be increased using the command:
+
+```bash
+oc describe StorageAutoScaler RESOURCE_NAME
path: alerts/openshift-container-storage-operator/StorageAutoScalingCapacityReached.md
user: login: parth-gr
body: can we create a direct command to get the status using jq maybe 
created_at: 2025-03-19T14:43:17Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,28 @@
+# StorageAutoScalingCapacityReached
+
+## Meaning
+
+The Storage Capacity Limit is the maximum storage that can be provisioned for the system. `StorageAutoScalingCapacityReached` alert is fired when the complete storage space has been utilized and hence additional storage cannot be allocated for the device class in the particular storage cluster.
+
+## Impact
+
+The storage cannot be scaled at the moment.
+
+## Diagnosis
+
+Retrieve the StorageAutoScaler Custom Resource for which the Storage Capacity is trying to be increased using the command:
+
+```bash
+oc describe StorageAutoScaler RESOURCE_NAME
+```
+If status of `storageCapacityLimitReached` is `true` then the storage cannot be scaled any further.
+
+## Mitigation
+
+- Scale storage: Depending on the type of cluster it will be needed to add
path: alerts/openshift-container-storage-operator/StorageAutoScalingCapacityReached.md
user: login: parth-gr
body: Remove this,

And add

```
oc patch storageautoscaler <resource> --type=merge --patch '{"spec":{"storageCapacityLimit":<capacity>}}'
```

add the capacity more than it is present

created_at: 2025-03-19T14:47:21Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,38 @@
+# StorageAutoScalingFailed
+
+## Meaning
+
+When scaling the system to accomodate additional storage,failures may occue due to various factors.In such instances, the `StorageAutoScalingFailed` alert is fired.
+
+## Impact
+
+The storage cannot be scaled at the moment.
+
+## Diagnosis
+
+There can be multiple reasons why the alert has been fired:
+
+- **AWS unable to match storage:**
+   In such scenarios, the first step is to detect the poubt if fauilure by examining the Persistent Volume(PV) and the Persistent Volume Claim (PVC) states. Check for any PVCs in `Pending` or `Bound` state using:
+   ```bash
+   oc describe pvc PVC_NAME
path: alerts/openshift-container-storage-operator/StorageAutoScalingFailed.md
user: login: parth-gr
body: Try a defined way how they can reach to pv and PVC name,

It should be based on the new osd  cr added, try to write kubectl cmds
created_at: 2025-03-19T14:48:44Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,38 @@
+# StorageAutoScalingFailed
+
+## Meaning
+
+When scaling the system to accomodate additional storage,failures may occue due to various factors.In such instances, the `StorageAutoScalingFailed` alert is fired.
+
+## Impact
+
+The storage cannot be scaled at the moment.
+
+## Diagnosis
+
+There can be multiple reasons why the alert has been fired:
+
+- **AWS unable to match storage:**
+   In such scenarios, the first step is to detect the poubt if fauilure by examining the Persistent Volume(PV) and the Persistent Volume Claim (PVC) states. Check for any PVCs in `Pending` or `Bound` state using:
+   ```bash
+   oc describe pvc PVC_NAME
+   ```
+
+   Check that the required permissions and volume types have been set correctly.
+   Additionally, list all PVs with `oc get pv` and verify that the pvcs are correctly recognised and mapped to the osds.
+
+- **OSD is not starting:**
+   ```bash
+   oc get pods -n <namespace>
+   ```
+
+   If the osd is stuck in "starting" stage, high chance the cluster resources might be exhausted. Check for the description of the osd pod.
+   ```bash
+   oc describe pod OSD_POD
+   ```
+
+## Mitigation
+
+- If AWS is unable to allocate storage, verify with the storage provider whether the utilized resources have reached their capacity limits.
+
+- Check upon the Ceph Storage: In case of issues with new OSDs, restart them and try again.
path: alerts/openshift-container-storage-operator/StorageAutoScalingFailed.md
user: login: parth-gr
body: also try to add try restarting rook operator,
also provide a cmd how to res-start rook operator 
created_at: 2025-03-19T14:49:39Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,38 @@
+# StorageAutoScalingFailed
+
+## Meaning
+
+When scaling the system to accomodate additional storage,failures may occue due to various factors.In such instances, the `StorageAutoScalingFailed` alert is fired.
+
+## Impact
+
+The storage cannot be scaled at the moment.
+
+## Diagnosis
+
+There can be multiple reasons why the alert has been fired:
+
+- **AWS unable to match storage:**
+   In such scenarios, the first step is to detect the poubt if fauilure by examining the Persistent Volume(PV) and the Persistent Volume Claim (PVC) states. Check for any PVCs in `Pending` or `Bound` state using:
+   ```bash
+   oc describe pvc PVC_NAME
+   ```
+
+   Check that the required permissions and volume types have been set correctly.
+   Additionally, list all PVs with `oc get pv` and verify that the pvcs are correctly recognised and mapped to the osds.
+
+- **OSD is not starting:**
+   ```bash
+   oc get pods -n <namespace>
+   ```
+
+   If the osd is stuck in "starting" stage, high chance the cluster resources might be exhausted. Check for the description of the osd pod.
+   ```bash
+   oc describe pod OSD_POD
+   ```
+
path: alerts/openshift-container-storage-operator/StorageAutoScalingFailed.md
user: login: parth-gr
body: the resource exhaustion mitigation is missing 
created_at: 2025-03-19T14:50:03Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,31 @@
+# StorageAutoScalingCapacityReached
+
+## Meaning
+
+The Storage Capacity Limit is the maximum storage that can be provisioned for the system. `StorageAutoScalingCapacityReached` alert is fired when the complete storage space has been utilized and hence additional storage cannot be allocated for the device class in the particular storage cluster.
+
+## Impact
+
+The storage cannot be auto-scaled at the moment.
+
+## Diagnosis
+
+Retrieve status of `storageCapacityLimitReached` from the StorageAutoScaler Custom Resource.
+
+```bash
+oc get StorageAutoScaler <resource_name> -o json | jq -r '.status.storageCapacityLimitReached'
+```
+
+If status of `storageCapacityLimitReached` is `true` then maximum capacity of the storage has been utilized.
+
+## Mitigation
+
+- Scale storage:
+  Increase the capacity by specifying it in the command below:
+  `bash
+    oc patch storageautoscaler <resource> --type=merge --patch '{"spec":{"storageCapacityLimit":<capacity>}}'
+    `
+
+- Delete information:
+  If not is possible to scale up the cluster it will be needed to delete
path: alerts/openshift-container-storage-operator/StorageAutoScalingCapacityReached.md
user: login: parth-gr
body: IMO, Not needed.
created_at: 2025-03-21T19:01:48Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,77 @@
+# StorageAutoScalingFailed
+
+## Meaning
+
+When scaling the system to accommodate additional storage, failures may occur due to various factors.In such instances, the `StorageAutoScalingFailed` alert is fired.
+
+## Impact
+
+The storage cannot be scaled at the moment.
+
+## Diagnosis
+
+There are multiple stages where the AWS storage configurations might not have matched the requested storage parameters.
path: alerts/openshift-container-storage-operator/StorageAutoScalingFailed.md
user: login: parth-gr
body: Dont be specific to AWS there are multiple cloud storage provider platforms
created_at: 2025-03-21T19:02:09Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,77 @@
+# StorageAutoScalingFailed
+
+## Meaning
+
+When scaling the system to accommodate additional storage, failures may occur due to various factors.In such instances, the `StorageAutoScalingFailed` alert is fired.
+
+## Impact
+
+The storage cannot be scaled at the moment.
+
+## Diagnosis
+
+There are multiple stages where the AWS storage configurations might not have matched the requested storage parameters.
+
+- OSD pod is in `Pending` state:
+  If it is in `Pending` state with the following logs, there is a high chance it is due to the exhaustion of the resources.
+
+  ```bash
+  Warning  FailedScheduling  3m53s  default-scheduler  0/6 nodes are available: 1 Insufficient cpu, 2 node(s) didn't match Pod's node affinity/selector, 3 node(s) had untolerated taint {node-role.kubernetes.io/master: }. preemption: 0/6 nodes are available: 1 Insufficient cpu, 5 Preemption is not helpful for scheduling.
+  ```
+
+  Describe the pod in `Pending` state and check for the events.
+
+  ```bash
+  oc describe pod <osd_pod>
+  ```
+
+  If there are no errors, retrieve the pvc of the pod in `Pending` state.
+
+  ```bash
+  oc get pod <osd_pod> -o jsonpath='{.spec.volumes[*].persistentVolumeClaim.claimName}'
path: alerts/openshift-container-storage-operator/StorageAutoScalingFailed.md
user: login: parth-gr
body: can you store it in a variable name and then populate it later
pvcName=$(oc get pod <osd_pod> -o jsonpath='{.spec.volumes[*].persistentVolumeClaim.claimName}')

oc get pvc $pvcName -o yaml

created_at: 2025-03-21T19:04:01Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,31 @@
+# StorageAutoScalingCapacityReached
+
+## Meaning
+
+The Storage Capacity Limit is the maximum storage that can be provisioned for the system. `StorageAutoScalingCapacityReached` alert is fired when the complete storage space has been utilized and hence additional storage cannot be allocated for the device class in the particular storage cluster.
+
+## Impact
+
+The storage cannot be auto-scaled at the moment.
+
+## Diagnosis
+
+Retrieve status of `storageCapacityLimitReached` from the StorageAutoScaler Custom Resource.
+
+```bash
+oc get StorageAutoScaler <resource_name> -o json | jq -r '.status.storageCapacityLimitReached'
path: alerts/openshift-container-storage-operator/StorageAutoScalingCapacityReached.md
user: login: parth-gr
body: namespace?
created_at: 2025-03-21T19:04:57Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,77 @@
+# StorageAutoScalingFailed
+
+## Meaning
+
+When scaling the system to accommodate additional storage, failures may occur due to various factors.In such instances, the `StorageAutoScalingFailed` alert is fired.
+
+## Impact
+
+The storage cannot be scaled at the moment.
+
+## Diagnosis
+
+There are multiple stages where the AWS storage configurations might not have matched the requested storage parameters.
+
+- OSD pod is in `Pending` state:
+  If it is in `Pending` state with the following logs, there is a high chance it is due to the exhaustion of the resources.
+
+  ```bash
path: alerts/openshift-container-storage-operator/StorageAutoScalingFailed.md
user: login: parth-gr
body: try increasing resources in this case 1.
created_at: 2025-03-21T19:06:25Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,77 @@
+# StorageAutoScalingFailed
+
+## Meaning
+
+When scaling the system to accommodate additional storage, failures may occur due to various factors.In such instances, the `StorageAutoScalingFailed` alert is fired.
+
+## Impact
+
+The storage cannot be scaled at the moment.
+
+## Diagnosis
+
+There are multiple stages where the AWS storage configurations might not have matched the requested storage parameters.
+
+- OSD pod is in `Pending` state:
+  If it is in `Pending` state with the following logs, there is a high chance it is due to the exhaustion of the resources.
+
+  ```bash
+  Warning  FailedScheduling  3m53s  default-scheduler  0/6 nodes are available: 1 Insufficient cpu, 2 node(s) didn't match Pod's node affinity/selector, 3 node(s) had untolerated taint {node-role.kubernetes.io/master: }. preemption: 0/6 nodes are available: 1 Insufficient cpu, 5 Preemption is not helpful for scheduling.
+  ```
+
+  Describe the pod in `Pending` state and check for the events.
+
+  ```bash
+  oc describe pod <osd_pod>
+  ```
+
+  If there are no errors, retrieve the pvc of the pod in `Pending` state.
+
+  ```bash
+  oc get pod <osd_pod> -o jsonpath='{.spec.volumes[*].persistentVolumeClaim.claimName}'
+  oc get pvc <pvc_name> -o yaml
+  ```
+
+  Check for the health and events of the PVC for detailed information.
+
+- PVC is in `Pending` state:
+
+  ```bash
+  oc get pvc <pvc_name> -o yaml
+  ```
+
+  Inspect the events of the pvc. Additionally, retrieve the pv of the pvc if it is still in `pending` state.
+
+  ```bash
+  oc get pv <pvc_name> -o jsonpath='{.spec.volumeName}'
+  ```
+
+- PV is in `Failed` state:
+  ```bash
+  oc get pv <pv_name> -o yaml
+  ```
+  If the pv is in `Failed` state, there might be a storage issue. Check the events of the pv for more information.
+
+## Mitigation
+
+- If the resources are exhausted and the mds pod is failing, the memory of the nodes needs to be increased:
path: alerts/openshift-container-storage-operator/StorageAutoScalingFailed.md
user: login: parth-gr
body: in general any pod
created_at: 2025-03-21T19:08:07Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,77 @@
+# StorageAutoScalingFailed
+
+## Meaning
+
+When scaling the system to accommodate additional storage, failures may occur due to various factors.In such instances, the `StorageAutoScalingFailed` alert is fired.
+
+## Impact
+
+The storage cannot be scaled at the moment.
+
+## Diagnosis
+
+There are multiple stages where the AWS storage configurations might not have matched the requested storage parameters.
+
+- OSD pod is in `Pending` state:
+  If it is in `Pending` state with the following logs, there is a high chance it is due to the exhaustion of the resources.
+
+  ```bash
+  Warning  FailedScheduling  3m53s  default-scheduler  0/6 nodes are available: 1 Insufficient cpu, 2 node(s) didn't match Pod's node affinity/selector, 3 node(s) had untolerated taint {node-role.kubernetes.io/master: }. preemption: 0/6 nodes are available: 1 Insufficient cpu, 5 Preemption is not helpful for scheduling.
+  ```
+
+  Describe the pod in `Pending` state and check for the events.
+
+  ```bash
+  oc describe pod <osd_pod>
+  ```
+
+  If there are no errors, retrieve the pvc of the pod in `Pending` state.
+
+  ```bash
+  oc get pod <osd_pod> -o jsonpath='{.spec.volumes[*].persistentVolumeClaim.claimName}'
+  oc get pvc <pvc_name> -o yaml
+  ```
+
+  Check for the health and events of the PVC for detailed information.
+
+- PVC is in `Pending` state:
+
+  ```bash
+  oc get pvc <pvc_name> -o yaml
+  ```
+
+  Inspect the events of the pvc. Additionally, retrieve the pv of the pvc if it is still in `pending` state.
+
+  ```bash
+  oc get pv <pvc_name> -o jsonpath='{.spec.volumeName}'
+  ```
+
+- PV is in `Failed` state:
+  ```bash
+  oc get pv <pv_name> -o yaml
+  ```
+  If the pv is in `Failed` state, there might be a storage issue. Check the events of the pv for more information.
path: alerts/openshift-container-storage-operator/StorageAutoScalingFailed.md
user: login: parth-gr
body: can we divided into case-1 cas-2? and separate mitigation for each case
created_at: 2025-03-21T19:08:46Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,77 @@
+# StorageAutoScalingFailed
+
+## Meaning
+
+When scaling the system to accommodate additional storage, failures may occur due to various factors.In such instances, the `StorageAutoScalingFailed` alert is fired.
+
+## Impact
+
+The storage cannot be scaled at the moment.
+
+## Diagnosis
+
+There are multiple stages where the AWS storage configurations might not have matched the requested storage parameters.
+
+- OSD pod is in `Pending` state:
+  If it is in `Pending` state with the following logs, there is a high chance it is due to the exhaustion of the resources.
+
+  ```bash
path: alerts/openshift-container-storage-operator/StorageAutoScalingFailed.md
user: login: ShravaniVangur
body: Yes this covers only the Diagnosis. I have added the suggestion part in Mitigation section.
created_at: 2025-03-24T06:35:00Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,77 @@
+# StorageAutoScalingFailed
+
+## Meaning
+
+When scaling the system to accommodate additional storage, failures may occur due to various factors.In such instances, the `StorageAutoScalingFailed` alert is fired.
+
+## Impact
+
+The storage cannot be scaled at the moment.
+
+## Diagnosis
+
+There are multiple stages where the AWS storage configurations might not have matched the requested storage parameters.
+
+- OSD pod is in `Pending` state:
+  If it is in `Pending` state with the following logs, there is a high chance it is due to the exhaustion of the resources.
+
+  ```bash
+  Warning  FailedScheduling  3m53s  default-scheduler  0/6 nodes are available: 1 Insufficient cpu, 2 node(s) didn't match Pod's node affinity/selector, 3 node(s) had untolerated taint {node-role.kubernetes.io/master: }. preemption: 0/6 nodes are available: 1 Insufficient cpu, 5 Preemption is not helpful for scheduling.
+  ```
+
+  Describe the pod in `Pending` state and check for the events.
+
+  ```bash
+  oc describe pod <osd_pod>
+  ```
+
+  If there are no errors, retrieve the pvc of the pod in `Pending` state.
+
+  ```bash
+  oc get pod <osd_pod> -o jsonpath='{.spec.volumes[*].persistentVolumeClaim.claimName}'
+  oc get pvc <pvc_name> -o yaml
+  ```
+
+  Check for the health and events of the PVC for detailed information.
+
+- PVC is in `Pending` state:
+
+  ```bash
+  oc get pvc <pvc_name> -o yaml
+  ```
+
+  Inspect the events of the pvc. Additionally, retrieve the pv of the pvc if it is still in `pending` state.
+
+  ```bash
+  oc get pv <pvc_name> -o jsonpath='{.spec.volumeName}'
+  ```
+
+- PV is in `Failed` state:
+  ```bash
+  oc get pv <pv_name> -o yaml
+  ```
+  If the pv is in `Failed` state, there might be a storage issue. Check the events of the pv for more information.
path: alerts/openshift-container-storage-operator/StorageAutoScalingFailed.md
user: login: ShravaniVangur
body: We are diagnosing the cause for each case and the cases for mitigation might get repetitive. Do you still want to add it for each case? 
created_at: 2025-03-24T06:48:10Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,77 @@
+# StorageAutoScalingFailed
+
+## Meaning
+
+When scaling the system to accommodate additional storage, failures may occur due to various factors.In such instances, the `StorageAutoScalingFailed` alert is fired.
+
+## Impact
+
+The storage cannot be scaled at the moment.
+
+## Diagnosis
+
+There are multiple stages where the AWS storage configurations might not have matched the requested storage parameters.
+
+- OSD pod is in `Pending` state:
+  If it is in `Pending` state with the following logs, there is a high chance it is due to the exhaustion of the resources.
+
+  ```bash
+  Warning  FailedScheduling  3m53s  default-scheduler  0/6 nodes are available: 1 Insufficient cpu, 2 node(s) didn't match Pod's node affinity/selector, 3 node(s) had untolerated taint {node-role.kubernetes.io/master: }. preemption: 0/6 nodes are available: 1 Insufficient cpu, 5 Preemption is not helpful for scheduling.
+  ```
+
+  Describe the pod in `Pending` state and check for the events.
+
+  ```bash
+  oc describe pod <osd_pod>
+  ```
+
+  If there are no errors, retrieve the pvc of the pod in `Pending` state.
+
+  ```bash
+  oc get pod <osd_pod> -o jsonpath='{.spec.volumes[*].persistentVolumeClaim.claimName}'
+  oc get pvc <pvc_name> -o yaml
+  ```
+
+  Check for the health and events of the PVC for detailed information.
+
+- PVC is in `Pending` state:
+
+  ```bash
+  oc get pvc <pvc_name> -o yaml
+  ```
+
+  Inspect the events of the pvc. Additionally, retrieve the pv of the pvc if it is still in `pending` state.
+
+  ```bash
+  oc get pv <pvc_name> -o jsonpath='{.spec.volumeName}'
+  ```
+
+- PV is in `Failed` state:
+  ```bash
+  oc get pv <pv_name> -o yaml
+  ```
+  If the pv is in `Failed` state, there might be a storage issue. Check the events of the pv for more information.
path: alerts/openshift-container-storage-operator/StorageAutoScalingFailed.md
user: login: parth-gr
body: We might consider different mitigation for different failures, so don't you think separating it out would be good, 
The scope of failures reason may also increase more in the future, that means descriptive and granular errors and their mitigations
created_at: 2025-03-24T06:58:14Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,30 @@
+# StorageAutoScalingCapacityReached
+
+## Meaning
+
+The Storage Capacity Limit is the maximum storage that can be provisioned for
+the system. `StorageAutoScalingCapacityReached` alert is fired when the complete
+storage space has been utilized and hence additional storage cannot be allocated
+for the device class in the particular storage cluster.
+
+## Impact
+
+The storage cannot be auto-scaled at the moment.
+
+## Diagnosis
+
+Retrieve status of `storageCapacityLimitReached` from the StorageAutoScaler CR.
+
+```bash
+oc get StorageAutoScaler <resource_name> -n openshift-storage -o json | jq -r '.status.storageCapacityLimitReached'
path: alerts/openshift-container-storage-operator/StorageAutoScalingCapacityReached.md
user: login: parth-gr
body: namespace
created_at: 2025-03-24T07:01:51Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,30 @@
+# StorageAutoScalingCapacityReached
+
+## Meaning
+
+The Storage Capacity Limit is the maximum storage that can be provisioned for
+the system. `StorageAutoScalingCapacityReached` alert is fired when the complete
+storage space has been utilized and hence additional storage cannot be allocated
+for the device class in the particular storage cluster.
+
+## Impact
+
+The storage cannot be auto-scaled at the moment.
+
+## Diagnosis
+
+Retrieve status of `storageCapacityLimitReached` from the StorageAutoScaler CR.
+
path: alerts/openshift-container-storage-operator/StorageAutoScalingCapacityReached.md
user: login: parth-gr
body: resource name and resource namespace would be provided by alert, maybe we want to specify or not specify it 
created_at: 2025-03-24T07:02:28Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,30 @@
+# StorageAutoScalingCapacityReached
+
+## Meaning
+
+The Storage Capacity Limit is the maximum storage that can be provisioned for
+the system. `StorageAutoScalingCapacityReached` alert is fired when the complete
+storage space has been utilized and hence additional storage cannot be allocated
+for the device class in the particular storage cluster.
+
+## Impact
+
+The storage cannot be auto-scaled at the moment.
+
+## Diagnosis
+
+Retrieve status of `storageCapacityLimitReached` from the StorageAutoScaler CR.
+
+```bash
+oc get StorageAutoScaler <resource_name> -n openshift-storage -o json | jq -r '.status.storageCapacityLimitReached'
+```
+
+If status of `storageCapacityLimitReached` is `true` then maximum capacity of
+the storage has been utilized.
+
+## Mitigation
+
+Scale storage: Increase the capacity by specifying it in the command below:
path: alerts/openshift-container-storage-operator/StorageAutoScalingCapacityReached.md
user: login: parth-gr
body: >Increase the capacity by specifying it in the command below:

Run the below command, specify the new increased StorageCapacityLimit 
created_at: 2025-03-24T07:03:59Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,30 @@
+# StorageAutoScalingCapacityReached
+
+## Meaning
+
+The Storage Capacity Limit is the maximum storage that can be provisioned for
+the system. `StorageAutoScalingCapacityReached` alert is fired when the complete
+storage space has been utilized and hence additional storage cannot be allocated
+for the device class in the particular storage cluster.
+
+## Impact
+
+The storage cannot be auto-scaled at the moment.
+
+## Diagnosis
+
+Retrieve status of `storageCapacityLimitReached` from the StorageAutoScaler CR.
+
+```bash
+oc get StorageAutoScaler <resource_name> -n openshift-storage -o json | jq -r '.status.storageCapacityLimitReached'
+```
+
+If status of `storageCapacityLimitReached` is `true` then maximum capacity of
+the storage has been utilized.
+
+## Mitigation
+
+Scale storage: Increase the capacity by specifying it in the command below:
+```bash
+oc patch storageautoscaler <resource> --type=merge --patch '{"spec":{"storageCapacityLimit":<capacity>}}'
path: alerts/openshift-container-storage-operator/StorageAutoScalingCapacityReached.md
user: login: parth-gr
body:  add 
 -n namespace
created_at: 2025-03-24T07:04:34Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,30 @@
+# StorageAutoScalingCapacityReached
+
+## Meaning
+
+The Storage Capacity Limit is the maximum storage that can be provisioned for
+the system. `StorageAutoScalingCapacityReached` alert is fired when the complete
+storage space has been utilized and hence additional storage cannot be allocated
+for the device class in the particular storage cluster.
+
+## Impact
+
+The storage cannot be auto-scaled at the moment.
+
+## Diagnosis
+
+Retrieve status of `storageCapacityLimitReached` from the StorageAutoScaler CR.
+
+```bash
+oc get StorageAutoScaler <resource_name> -n openshift-storage -o json | jq -r '.status.storageCapacityLimitReached'
+```
+
+If status of `storageCapacityLimitReached` is `true` then maximum capacity of
+the storage has been utilized.
+
+## Mitigation
+
+Scale storage: Increase the capacity by specifying it in the command below:
path: alerts/openshift-container-storage-operator/StorageAutoScalingCapacityReached.md
user: login: parth-gr
body: Note: 
`status.error.message` will be specifying the min required increased StorageCapacityLimit, so either at least specify that or more
created_at: 2025-03-24T07:08:27Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,77 @@
+# StorageAutoScalingFailed
+
+## Meaning
+
+When scaling the system to accommodate additional storage, failures may occur
+due to various factors.In such instances, the `StorageAutoScalingFailed` alert
path: alerts/openshift-container-storage-operator/StorageAutoScalingFailed.md
user: login: parth-gr
body: add a space,  
factor. In
created_at: 2025-03-24T07:08:57Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,77 @@
+# StorageAutoScalingFailed
+
+## Meaning
+
+When scaling the system to accommodate additional storage, failures may occur
+due to various factors.In such instances, the `StorageAutoScalingFailed` alert
+is fired.
+
+## Impact
+
+The storage cannot be scaled at the moment.
+
+## Diagnosis
+
+There are multiple stages where the storage configurations might not have
+matched the requested storage parameters.
+
+1. OSD pod is in `Pending` state:
+    If it is in `Pending` state with the following logs, there is a high chance
+    it is due to the exhaustion of the resources.
+    ```bash
+    Warning  FailedScheduling  3m53s  default-scheduler  0/6 nodes are available: 1 Insufficient cpu, 2 node(s) didn't match Pod's node affinity/selector, 3 node(s) had untolerated taint {node-role.kubernetes.io/master: }. preemption: 0/6 nodes are available: 1 Insufficient cpu, 5 Preemption is not helpful for scheduling.
+    ```
+
+    Describe the pod in `Pending` state and check for the events.
+    ```bash
+    oc describe pod <osd_pod>
+    ```
+
+    If there are no errors, retrieve the pvc of the pod in `Pending` state.
+    ```bash
+    pvcName=$(oc get pod <osd_pod> -o jsonpath='{.spec.volumes[*].persistentVolumeClaim.claimName}')
+    oc get pvc $pvcName -o yaml
+    ```
+    Check for the health and events of the PVC for detailed information.
+
+2. PVC is in `Pending` state:
+
+    ```bash
+    oc get pvc <pvc_name> -o yaml
+    ```
+
+    Inspect the events of the pvc. Additionally, retrieve the pv of the pvc if
+    it is still in `pending` state.
+    ```bash
+    oc get pv <pvc_name> -o jsonpath='{.spec.volumeName}'
+    ```
+
+3. PV is in `Failed` state:
+    ```bash
+    oc get pv <pv_name> -o yaml
+    ```
+    If the pv is in `Failed` state, there might be a storage issue. Check the
+    events of the pv for more information.
+
+## Mitigation
+
+1. If the resources are exhausted and pods are failing, the memory of the nodes
+needs to be increased:
+    ```bash
path: alerts/openshift-container-storage-operator/StorageAutoScalingFailed.md
user: login: parth-gr
body: earlier the mitigation was correct, it should be specific to pod not autoscaler
created_at: 2025-03-24T07:10:24Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,77 @@
+# StorageAutoScalingFailed
+
+## Meaning
+
+When scaling the system to accommodate additional storage, failures may occur
+due to various factors.In such instances, the `StorageAutoScalingFailed` alert
+is fired.
+
+## Impact
+
+The storage cannot be scaled at the moment.
+
+## Diagnosis
+
+There are multiple stages where the storage configurations might not have
+matched the requested storage parameters.
+
+1. OSD pod is in `Pending` state:
+    If it is in `Pending` state with the following logs, there is a high chance
+    it is due to the exhaustion of the resources.
+    ```bash
+    Warning  FailedScheduling  3m53s  default-scheduler  0/6 nodes are available: 1 Insufficient cpu, 2 node(s) didn't match Pod's node affinity/selector, 3 node(s) had untolerated taint {node-role.kubernetes.io/master: }. preemption: 0/6 nodes are available: 1 Insufficient cpu, 5 Preemption is not helpful for scheduling.
+    ```
+
+    Describe the pod in `Pending` state and check for the events.
+    ```bash
+    oc describe pod <osd_pod>
+    ```
+
+    If there are no errors, retrieve the pvc of the pod in `Pending` state.
+    ```bash
+    pvcName=$(oc get pod <osd_pod> -o jsonpath='{.spec.volumes[*].persistentVolumeClaim.claimName}')
+    oc get pvc $pvcName -o yaml
+    ```
+    Check for the health and events of the PVC for detailed information.
+
+2. PVC is in `Pending` state:
+
+    ```bash
+    oc get pvc <pvc_name> -o yaml
+    ```
+
+    Inspect the events of the pvc. Additionally, retrieve the pv of the pvc if
+    it is still in `pending` state.
+    ```bash
+    oc get pv <pvc_name> -o jsonpath='{.spec.volumeName}'
+    ```
+
+3. PV is in `Failed` state:
+    ```bash
+    oc get pv <pv_name> -o yaml
+    ```
+    If the pv is in `Failed` state, there might be a storage issue. Check the
+    events of the pv for more information.
+
+## Mitigation
+
+1. If the resources are exhausted and pods are failing, the memory of the nodes
+needs to be increased:
+    ```bash
+    oc patch storageautoscaler <resource> --type=merge --patch '{"spec":{"storageCapacityLimit":<capacity>}}'
+    ```
+
+2. Alternatively, one could try to restart the rook operator:
+    ```bash
+    oc rollout restart deployment rook-ceph-operator
+    ```
+
+    After restarting, look for errors or warnings in the logs of rook-ceph-operator:
+    ```bash
+    oc logs -l app=rook-ceph-operator
+    ```
+
+    And monitor the status of the pods and cephcluster post restart.
+
+3. Check upon the Ceph Storage: In case of issues with new OSDs, restart them
path: alerts/openshift-container-storage-operator/StorageAutoScalingFailed.md
user: login: parth-gr
body: also try to add in case of pv in pending state, try to look at pv errros and try to fix the 3rd party storage provider
created_at: 2025-03-24T07:11:13Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,34 @@
+# StorageAutoScalingCapacityReached
+
+## Meaning
+
+The Storage Capacity Limit is the maximum storage that can be provisioned for
+the system. `StorageAutoScalingCapacityReached` alert is fired when the complete
+storage space has been utilized and hence additional storage cannot be allocated
+for the device class in the particular storage cluster.
+
+## Impact
+
+The storage cannot be auto-scaled at the moment.
+
+## Diagnosis
+
+Retrieve status of `storageCapacityLimitReached` from the StorageAutoScaler CR.
+The `resource_name` and `resource_namespace` would be specified in the alert itself.
+
+```bash
+oc get StorageAutoScaler <resource_name> -n <resource_namespace> -o json | jq -r '.status.storageCapacityLimitReached'
+```
+
+If status of `storageCapacityLimitReached` is `true` then maximum capacity of
+the storage has been utilized.
+
+## Mitigation
+
+Scale storage: Execute the following command with the updated `storageCapacityLimit`-
+```bash
+oc patch storageautoscaler <resource> -n <namespace> --type=merge --patch '{"spec":{"storageCapacityLimit":<capacity>}}'
path: alerts/openshift-container-storage-operator/StorageAutoScalingCapacityReached.md
user: login: parth-gr
body: oc patch storageautoscaler <resource> -n <namespace> --type=merge --patch '{"spec":{"storageCapacityLimit":<storageCapacityLimit>}}'
created_at: 2025-03-24T12:15:59Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,83 @@
+# StorageAutoScalingFailed
+
+## Meaning
+
+When scaling the system to accommodate additional storage, failures may occur
path: alerts/openshift-container-storage-operator/StorageAutoScalingFailed.md
user: login: parth-gr
body: During automatic storage scaling
created_at: 2025-03-24T12:16:37Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,83 @@
+# StorageAutoScalingFailed
+
+## Meaning
+
+When scaling the system to accommodate additional storage, failures may occur
+due to various factors. In such instances, the `StorageAutoScalingFailed` alert
+is fired.
+
+## Impact
+
+The storage cannot be scaled at the moment.
+
+## Diagnosis
+
+There are multiple stages where the storage configurations might not have
+matched the requested storage parameters.
+
+1. OSD pod is in `Pending` state:
+    If it is in `Pending` state with the following logs, there is a high chance
+    it is due to the exhaustion of the resources.
+    ```bash
+    Warning  FailedScheduling  3m53s  default-scheduler  0/6 nodes are available: 1 Insufficient cpu, 2 node(s) didn't match Pod's node affinity/selector, 3 node(s) had untolerated taint {node-role.kubernetes.io/master: }. preemption: 0/6 nodes are available: 1 Insufficient cpu, 5 Preemption is not helpful for scheduling.
+    ```
+
+    Describe the pod in `Pending` state and check for the events.
+    ```bash
+    oc describe pod <osd_pod>
+    ```
+
+    If there are no errors, retrieve the pvc of the pod in `Pending` state.
+    ```bash
+    pvcName=$(oc get pod <osd_pod> -o jsonpath='{.spec.volumes[*].persistentVolumeClaim.claimName}')
+    oc get pvc $pvcName -o yaml
+    ```
+    Check for the health and events of the PVC for detailed information.
+
+2. PVC is in `Pending` state:
+
+    ```bash
+    oc get pvc <pvc_name> -o yaml
+    ```
+
+    Inspect the events of the pvc. Additionally, retrieve the pv of the pvc if
+    it is still in `pending` state.
+    ```bash
+    oc get pv <pvc_name> -o jsonpath='{.spec.volumeName}'
+    ```
+
+3. PV is in `Failed` state:
+    ```bash
+    oc get pv <pv_name> -o yaml
+    ```
+    If the pv is in `Failed` state, there might be a storage issue. Check the
+    events of the pv for more information.
+
+## Mitigation
+
+1. If the resources are exhausted and pods are failing, the memory of the nodes
+needs to be increased:
+    ```bash
+    oc patch storageautoscaler <resource> --type=merge --patch '{"spec":{"storageCapacityLimit":<capacity>}}'
path: alerts/openshift-container-storage-operator/StorageAutoScalingFailed.md
user: login: parth-gr
body: we need to add more resources at storagecluster 
created_at: 2025-03-24T12:19:48Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,34 @@
+# StorageAutoScalingCapacityReached
+
+## Meaning
+
+The Storage Capacity Limit is the maximum storage that can be provisioned for
+the system. `StorageAutoScalingCapacityReached` alert is fired when the complete
+storage space has been utilized and hence additional storage cannot be allocated
+for the device class in the particular storage cluster.
+
+## Impact
+
+The storage cannot be auto-scaled at the moment.
+
+## Diagnosis
+
+Retrieve status of `storageCapacityLimitReached` from the StorageAutoScaler CR.
+The `resource_name` and `resource_namespace` would be specified in the alert itself.
+
+```bash
+oc get StorageAutoScaler <resource_name> -n <resource_namespace> -o json | jq -r '.status.storageCapacityLimitReached'
+```
+
+If status of `storageCapacityLimitReached` is `true` then maximum capacity of
+the storage has been utilized.
+
+## Mitigation
+
+Scale storage: Execute the following command with the updated `storageCapacityLimit`-
+```bash
+oc patch storageautoscaler -n --type=merge --patch '{"spec":{"storageCapacityLimit":}}'
path: alerts/openshift-container-storage-operator/StorageAutoScalingCapacityReached.md
user: login: Madhu-1
body: ```suggestion
oc patch storageautoscaler <resource_name> -n <resource_namespace> --type=merge --patch '{"spec":{"storageCapacityLimit":}}'
```
created_at: 2025-03-24T12:54:50Z
author_association: NONE
diff_hunk: @@ -0,0 +1,34 @@
+# StorageAutoScalingCapacityReached
+
+## Meaning
+
+The Storage Capacity Limit is the maximum storage that can be provisioned for
+the system. `StorageAutoScalingCapacityReached` alert is fired when the complete
+storage space has been utilized and hence additional storage cannot be allocated
+for the device class in the particular storage cluster.
+
+## Impact
+
+The storage cannot be auto-scaled at the moment.
+
+## Diagnosis
+
+Retrieve status of `storageCapacityLimitReached` from the StorageAutoScaler CR.
+The `resource_name` and `resource_namespace` would be specified in the alert itself.
+
+```bash
+oc get StorageAutoScaler <resource_name> -n <resource_namespace> -o json | jq -r '.status.storageCapacityLimitReached'
path: alerts/openshift-container-storage-operator/StorageAutoScalingCapacityReached.md
user: login: Madhu-1
body: ```suggestion
oc get storageautoscaler <resource_name> -n <resource_namespace> -o json | jq -r '.status.storageCapacityLimitReached'
```
created_at: 2025-03-24T12:55:05Z
author_association: NONE
diff_hunk: @@ -0,0 +1,82 @@
+# StorageAutoScalingFailed
+
+## Meaning
+
+During automatic storage scaling, failures may occur due to various factors. In
+such instances, the `StorageAutoScalingFailed` alert is fired.
+
+## Impact
+
+The storage cannot be scaled at the moment.
+
+## Diagnosis
+
+There are multiple stages where the storage configurations might not have
+matched the requested storage parameters.
path: alerts/openshift-container-storage-operator/StorageAutoScalingFailed.md
user: login: Madhu-1
body: ```suggestion
There are multiple stages where the storage configurations might not have matched the requested storage parameters.
```
created_at: 2025-03-24T12:55:44Z
author_association: NONE
diff_hunk: @@ -0,0 +1,82 @@
+# StorageAutoScalingFailed
+
+## Meaning
+
+During automatic storage scaling, failures may occur due to various factors. In
+such instances, the `StorageAutoScalingFailed` alert is fired.
+
+## Impact
+
+The storage cannot be scaled at the moment.
+
+## Diagnosis
+
+There are multiple stages where the storage configurations might not have
+matched the requested storage parameters.
+
+1. OSD pod is in `Pending` state:
+    If it is in `Pending` state with the following logs, there is a high chance
+    it is due to the exhaustion of the resources.
+    ```bash
+    Warning  FailedScheduling  3m53s  default-scheduler  0/6 nodes are available: 1 Insufficient cpu, 2 node(s) didn't match Pod's node affinity/selector, 3 node(s) had untolerated taint {node-role.kubernetes.io/master: }. preemption: 0/6 nodes are available: 1 Insufficient cpu, 5 Preemption is not helpful for scheduling.
+    ```
+
+    Describe the pod in `Pending` state and check for the events.
+    ```bash
+    oc describe pod <osd_pod>
+    ```
+
+    If there are no errors, retrieve the pvc of the pod in `Pending` state.
+    ```bash
+    pvcName=$(oc get pod <osd_pod> -o jsonpath='{.spec.volumes[*].persistentVolumeClaim.claimName}')
path: alerts/openshift-container-storage-operator/StorageAutoScalingFailed.md
user: login: Madhu-1
body: add namespace to all oc commands or ask admin to switch to project
created_at: 2025-03-24T12:57:05Z
author_association: NONE
diff_hunk: @@ -0,0 +1,82 @@
+# StorageAutoScalingFailed
+
+## Meaning
+
+During automatic storage scaling, failures may occur due to various factors. In
+such instances, the `StorageAutoScalingFailed` alert is fired.
+
+## Impact
+
+The storage cannot be scaled at the moment.
+
+## Diagnosis
+
+There are multiple stages where the storage configurations might not have
+matched the requested storage parameters.
+
+1. OSD pod is in `Pending` state:
+    If it is in `Pending` state with the following logs, there is a high chance
+    it is due to the exhaustion of the resources.
+    ```bash
+    Warning  FailedScheduling  3m53s  default-scheduler  0/6 nodes are available: 1 Insufficient cpu, 2 node(s) didn't match Pod's node affinity/selector, 3 node(s) had untolerated taint {node-role.kubernetes.io/master: }. preemption: 0/6 nodes are available: 1 Insufficient cpu, 5 Preemption is not helpful for scheduling.
+    ```
+
+    Describe the pod in `Pending` state and check for the events.
+    ```bash
+    oc describe pod <osd_pod>
+    ```
+
+    If there are no errors, retrieve the pvc of the pod in `Pending` state.
+    ```bash
+    pvcName=$(oc get pod <osd_pod> -o jsonpath='{.spec.volumes[*].persistentVolumeClaim.claimName}')
+    oc get pvc $pvcName -o yaml
+    ```
+    Check for the health and events of the PVC for detailed information.
+
+2. PVC is in `Pending` state:
+
+    ```bash
+    oc get pvc <pvc_name> -o yaml
+    ```
+
+    Inspect the events of the pvc. Additionally, retrieve the pv of the pvc if
+    it is still in `pending` state.
+    ```bash
+    oc get pv <pvc_name> -o jsonpath='{.spec.volumeName}'
path: alerts/openshift-container-storage-operator/StorageAutoScalingFailed.md
user: login: Madhu-1
body: ```suggestion
    oc get pvc <pvc_name> -o jsonpath='{.spec.volumeName}'
```
created_at: 2025-03-24T12:57:55Z
author_association: NONE
diff_hunk: @@ -0,0 +1,82 @@
+# StorageAutoScalingFailed
+
+## Meaning
+
+During automatic storage scaling, failures may occur due to various factors. In
+such instances, the `StorageAutoScalingFailed` alert is fired.
+
+## Impact
+
+The storage cannot be scaled at the moment.
+
+## Diagnosis
+
+There are multiple stages where the storage configurations might not have
+matched the requested storage parameters.
+
+1. OSD pod is in `Pending` state:
+    If it is in `Pending` state with the following logs, there is a high chance
+    it is due to the exhaustion of the resources.
+    ```bash
+    Warning  FailedScheduling  3m53s  default-scheduler  0/6 nodes are available: 1 Insufficient cpu, 2 node(s) didn't match Pod's node affinity/selector, 3 node(s) had untolerated taint {node-role.kubernetes.io/master: }. preemption: 0/6 nodes are available: 1 Insufficient cpu, 5 Preemption is not helpful for scheduling.
+    ```
+
+    Describe the pod in `Pending` state and check for the events.
+    ```bash
+    oc describe pod <osd_pod>
+    ```
+
+    If there are no errors, retrieve the pvc of the pod in `Pending` state.
+    ```bash
+    pvcName=$(oc get pod <osd_pod> -o jsonpath='{.spec.volumes[*].persistentVolumeClaim.claimName}')
+    oc get pvc $pvcName -o yaml
+    ```
+    Check for the health and events of the PVC for detailed information.
+
+2. PVC is in `Pending` state:
+
+    ```bash
+    oc get pvc <pvc_name> -o yaml
+    ```
+
+    Inspect the events of the pvc. Additionally, retrieve the pv of the pvc if
+    it is still in `pending` state.
+    ```bash
+    oc get pv <pvc_name> -o jsonpath='{.spec.volumeName}'
+    ```
+
+3. PV is in `Failed` state:
+    ```bash
+    oc get pv <pv_name> -o yaml
+    ```
+    If the pv is in `Failed` state, there might be a storage issue. Check the
+    events of the pv for more information.
+
+## Mitigation
+
+1. If resources are exhausted and pods are failing, additional resources need to
+be allocated to the storage cluster:
+    ```bash
+    oc patch storageautoscaler -n --type=merge --patch '{"spec":{"storageCapacityLimit":}}'
+    ```
+
+2. Alternatively, one could try to restart the rook operator:
path: alerts/openshift-container-storage-operator/StorageAutoScalingFailed.md
user: login: Madhu-1
body: why Rook operator restart is required?
created_at: 2025-03-24T12:58:32Z
author_association: NONE
diff_hunk: @@ -0,0 +1,82 @@
+# StorageAutoScalingFailed
+
+## Meaning
+
+During automatic storage scaling, failures may occur due to various factors. In
+such instances, the `StorageAutoScalingFailed` alert is fired.
+
+## Impact
+
+The storage cannot be scaled at the moment.
+
+## Diagnosis
+
+There are multiple stages where the storage configurations might not have
+matched the requested storage parameters.
+
+1. OSD pod is in `Pending` state:
+    If it is in `Pending` state with the following logs, there is a high chance
+    it is due to the exhaustion of the resources.
+    ```bash
+    Warning  FailedScheduling  3m53s  default-scheduler  0/6 nodes are available: 1 Insufficient cpu, 2 node(s) didn't match Pod's node affinity/selector, 3 node(s) had untolerated taint {node-role.kubernetes.io/master: }. preemption: 0/6 nodes are available: 1 Insufficient cpu, 5 Preemption is not helpful for scheduling.
+    ```
+
+    Describe the pod in `Pending` state and check for the events.
+    ```bash
+    oc describe pod <osd_pod>
+    ```
+
+    If there are no errors, retrieve the pvc of the pod in `Pending` state.
+    ```bash
+    pvcName=$(oc get pod <osd_pod> -o jsonpath='{.spec.volumes[*].persistentVolumeClaim.claimName}')
+    oc get pvc $pvcName -o yaml
+    ```
+    Check for the health and events of the PVC for detailed information.
+
+2. PVC is in `Pending` state:
+
+    ```bash
+    oc get pvc <pvc_name> -o yaml
+    ```
+
+    Inspect the events of the pvc. Additionally, retrieve the pv of the pvc if
+    it is still in `pending` state.
+    ```bash
+    oc get pv <pvc_name> -o jsonpath='{.spec.volumeName}'
+    ```
+
+3. PV is in `Failed` state:
+    ```bash
+    oc get pv <pv_name> -o yaml
+    ```
+    If the pv is in `Failed` state, there might be a storage issue. Check the
+    events of the pv for more information.
+
+## Mitigation
+
+1. If resources are exhausted and pods are failing, additional resources need to
+be allocated to the storage cluster:
+    ```bash
+    oc patch storageautoscaler -n --type=merge --patch '{"spec":{"storageCapacityLimit":}}'
+    ```
+
+2. Alternatively, one could try to restart the rook operator:
path: alerts/openshift-container-storage-operator/StorageAutoScalingFailed.md
user: login: parth-gr
body: In vertical scaling,
If the pv takes time to resize, the osd won't get resize correctly, 

1) Restart osd pods
2) Restart the rook operator

This PR https://github.com/rook/rook/pull/15251 should fix it
cc @travis for more thoughts
created_at: 2025-03-24T13:10:27Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,34 @@
+# StorageAutoScalingCapacityReached
+
+## Meaning
+
+The Storage Capacity Limit is the maximum storage that can be provisioned for
+the system. `StorageAutoScalingCapacityReached` alert is fired when the complete
+storage space has been utilized and hence additional storage cannot be allocated
+for the device class in the particular storage cluster.
+
+## Impact
+
+The storage cannot be auto-scaled at the moment.
+
+## Diagnosis
+
+Retrieve status of `storageCapacityLimitReached` from the StorageAutoScaler CR.
+The `resource_name` and `resource_namespace` would be specified in the alert itself.
+
+```bash
+oc get storageautoscaler <resource_name> -n <resource_namespace> -o json | jq -r '.status.storageCapacityLimitReached'
+```
+
+If status of `storageCapacityLimitReached` is `true` then maximum capacity of
+the storage has been utilized.
+
+## Mitigation
path: alerts/openshift-container-storage-operator/StorageAutoScalingCapacityReached.md
user: login: travisn
body: Another possible mitigation should be to disable the auto scale, not just to increase the limit. The user needs to decide what they prefer. There are ways to scale out (add more OSDs) without auto-scale enabled. 
created_at: 2025-03-24T18:03:46Z
author_association: NONE
diff_hunk: @@ -0,0 +1,84 @@
+# StorageAutoScalingFailed
+
+## Meaning
+
+During automatic storage scaling, failures may occur due to various factors. In
+such instances, the `StorageAutoScalingFailed` alert is fired.
+
+## Impact
+
+The storage cannot be scaled at the moment.
+
+## Diagnosis
+
+There are multiple stages where the storage configurations might not have matched
+the requested storage parameters.
+
+1. OSD pod is in `Pending` state:
+    If it is in `Pending` state with the following logs, there is a high chance
+    it is due to the exhaustion of the resources.
+    ```bash
+    Warning  FailedScheduling  3m53s  default-scheduler  0/6 nodes are available: 1 Insufficient cpu, 2 node(s) didn't match Pod's node affinity/selector, 3 node(s) had untolerated taint {node-role.kubernetes.io/master: }. preemption: 0/6 nodes are available: 1 Insufficient cpu, 5 Preemption is not helpful for scheduling.
+    ```
+
+    Describe the pod in `Pending` state and check for the events.
+    ```bash
+    oc describe pod <osd_pod> -n <namespace>
+    ```
+
+    If there are no errors, retrieve the pvc of the pod in `Pending` state.
+    ```bash
+    pvcName=$(oc get pod <osd_pod> -n <namespace> -o jsonpath='{.spec.volumes[*].persistentVolumeClaim.claimName}')
+    oc get pvc $pvcName -n <namespace> -o yaml
+    ```
+    Check for the health and events of the PVC for detailed information.
+
+2. PVC is in `Pending` state:
+
+    ```bash
+    oc get pvc <pvc_name> -n <namespace> -o yaml
+    ```
+
+    Inspect the events of the pvc. Additionally, retrieve the pv of the pvc if
+    it is still in `pending` state.
+    ```bash
+    oc get pvc <pvc_name> -n <namespace> -o jsonpath='{.spec.volumeName}'
+    ```
+
+3. PV is in `Failed` state:
+    ```bash
+    oc get pv <pv_name> -n <namespace> -o yaml
+    ```
+    If the pv is in `Failed` state, there might be a storage issue. Check the
+    events of the pv for more information.
+
+## Mitigation
+
+1. If resources are exhausted and pods are failing, additional resources need to
+be allocated to the storage cluster:
path: alerts/openshift-container-storage-operator/StorageAutoScalingFailed.md
user: login: travisn
body: Changing to the performance profile actually will not help resolve this situation. Enabling performance profile will actually make the problem worse since even more resources will be requested for ODF pods. Instead, the mitigation should be to add more nodes, or increase the cpu and/or memory on the existing nodes. 
created_at: 2025-03-24T18:07:05Z
author_association: NONE
diff_hunk: @@ -0,0 +1,84 @@
+# StorageAutoScalingFailed
+
+## Meaning
+
+During automatic storage scaling, failures may occur due to various factors. In
+such instances, the `StorageAutoScalingFailed` alert is fired.
+
+## Impact
+
+The storage cannot be scaled at the moment.
+
+## Diagnosis
+
+There are multiple stages where the storage configurations might not have matched
+the requested storage parameters.
+
+1. OSD pod is in `Pending` state:
+    If it is in `Pending` state with the following logs, there is a high chance
+    it is due to the exhaustion of the resources.
+    ```bash
+    Warning  FailedScheduling  3m53s  default-scheduler  0/6 nodes are available: 1 Insufficient cpu, 2 node(s) didn't match Pod's node affinity/selector, 3 node(s) had untolerated taint {node-role.kubernetes.io/master: }. preemption: 0/6 nodes are available: 1 Insufficient cpu, 5 Preemption is not helpful for scheduling.
+    ```
+
+    Describe the pod in `Pending` state and check for the events.
+    ```bash
+    oc describe pod <osd_pod> -n <namespace>
+    ```
+
+    If there are no errors, retrieve the pvc of the pod in `Pending` state.
+    ```bash
+    pvcName=$(oc get pod <osd_pod> -n <namespace> -o jsonpath='{.spec.volumes[*].persistentVolumeClaim.claimName}')
+    oc get pvc $pvcName -n <namespace> -o yaml
+    ```
+    Check for the health and events of the PVC for detailed information.
+
+2. PVC is in `Pending` state:
+
+    ```bash
+    oc get pvc <pvc_name> -n <namespace> -o yaml
+    ```
+
+    Inspect the events of the pvc. Additionally, retrieve the pv of the pvc if
+    it is still in `pending` state.
+    ```bash
+    oc get pvc <pvc_name> -n <namespace> -o jsonpath='{.spec.volumeName}'
+    ```
+
+3. PV is in `Failed` state:
+    ```bash
+    oc get pv <pv_name> -n <namespace> -o yaml
+    ```
+    If the pv is in `Failed` state, there might be a storage issue. Check the
+    events of the pv for more information.
+
+## Mitigation
+
+1. If resources are exhausted and pods are failing, additional resources need to
+be allocated to the storage cluster:
+    ```bash
+    oc patch -n openshift-storage storagecluster ocs-storagecluster \
+    --type merge \
+    --patch '{"spec": {"resourceProfile"="performance"}}'
+    ```
+
+2. Check upon the Ceph Storage: In case of issues with new OSDs, restart them
path: alerts/openshift-container-storage-operator/StorageAutoScalingFailed.md
user: login: travisn
body: This is a vague statement. When can the user detect restarting the OSDs is needed?
created_at: 2025-03-24T18:08:34Z
author_association: NONE
diff_hunk: @@ -0,0 +1,84 @@
+# StorageAutoScalingFailed
+
+## Meaning
+
+During automatic storage scaling, failures may occur due to various factors. In
+such instances, the `StorageAutoScalingFailed` alert is fired.
+
+## Impact
+
+The storage cannot be scaled at the moment.
+
+## Diagnosis
+
+There are multiple stages where the storage configurations might not have matched
+the requested storage parameters.
+
+1. OSD pod is in `Pending` state:
+    If it is in `Pending` state with the following logs, there is a high chance
+    it is due to the exhaustion of the resources.
+    ```bash
+    Warning  FailedScheduling  3m53s  default-scheduler  0/6 nodes are available: 1 Insufficient cpu, 2 node(s) didn't match Pod's node affinity/selector, 3 node(s) had untolerated taint {node-role.kubernetes.io/master: }. preemption: 0/6 nodes are available: 1 Insufficient cpu, 5 Preemption is not helpful for scheduling.
+    ```
+
+    Describe the pod in `Pending` state and check for the events.
+    ```bash
+    oc describe pod <osd_pod> -n <namespace>
+    ```
+
+    If there are no errors, retrieve the pvc of the pod in `Pending` state.
+    ```bash
+    pvcName=$(oc get pod <osd_pod> -n <namespace> -o jsonpath='{.spec.volumes[*].persistentVolumeClaim.claimName}')
+    oc get pvc $pvcName -n <namespace> -o yaml
+    ```
+    Check for the health and events of the PVC for detailed information.
+
+2. PVC is in `Pending` state:
+
+    ```bash
+    oc get pvc <pvc_name> -n <namespace> -o yaml
+    ```
+
+    Inspect the events of the pvc. Additionally, retrieve the pv of the pvc if
+    it is still in `pending` state.
+    ```bash
+    oc get pvc <pvc_name> -n <namespace> -o jsonpath='{.spec.volumeName}'
+    ```
+
+3. PV is in `Failed` state:
+    ```bash
+    oc get pv <pv_name> -n <namespace> -o yaml
+    ```
+    If the pv is in `Failed` state, there might be a storage issue. Check the
+    events of the pv for more information.
+
+## Mitigation
+
+1. If resources are exhausted and pods are failing, additional resources need to
+be allocated to the storage cluster:
+    ```bash
+    oc patch -n openshift-storage storagecluster ocs-storagecluster \
+    --type merge \
+    --patch '{"spec": {"resourceProfile"="performance"}}'
+    ```
+
+2. Check upon the Ceph Storage: In case of issues with new OSDs, restart them
+and try again.
+
+3. Alternatively, one could try to restart the rook operator:
path: alerts/openshift-container-storage-operator/StorageAutoScalingFailed.md
user: login: travisn
body: When to restart the rook operator, and what problem will it resolve?  
created_at: 2025-03-24T18:09:00Z
author_association: NONE
diff_hunk: @@ -0,0 +1,84 @@
+# StorageAutoScalingFailed
+
+## Meaning
+
+During automatic storage scaling, failures may occur due to various factors. In
+such instances, the `StorageAutoScalingFailed` alert is fired.
+
+## Impact
+
+The storage cannot be scaled at the moment.
+
+## Diagnosis
+
+There are multiple stages where the storage configurations might not have matched
+the requested storage parameters.
+
+1. OSD pod is in `Pending` state:
+    If it is in `Pending` state with the following logs, there is a high chance
+    it is due to the exhaustion of the resources.
+    ```bash
+    Warning  FailedScheduling  3m53s  default-scheduler  0/6 nodes are available: 1 Insufficient cpu, 2 node(s) didn't match Pod's node affinity/selector, 3 node(s) had untolerated taint {node-role.kubernetes.io/master: }. preemption: 0/6 nodes are available: 1 Insufficient cpu, 5 Preemption is not helpful for scheduling.
+    ```
+
+    Describe the pod in `Pending` state and check for the events.
+    ```bash
+    oc describe pod <osd_pod> -n <namespace>
+    ```
+
+    If there are no errors, retrieve the pvc of the pod in `Pending` state.
+    ```bash
+    pvcName=$(oc get pod <osd_pod> -n <namespace> -o jsonpath='{.spec.volumes[*].persistentVolumeClaim.claimName}')
+    oc get pvc $pvcName -n <namespace> -o yaml
+    ```
+    Check for the health and events of the PVC for detailed information.
+
+2. PVC is in `Pending` state:
+
+    ```bash
+    oc get pvc <pvc_name> -n <namespace> -o yaml
+    ```
+
+    Inspect the events of the pvc. Additionally, retrieve the pv of the pvc if
+    it is still in `pending` state.
+    ```bash
+    oc get pvc <pvc_name> -n <namespace> -o jsonpath='{.spec.volumeName}'
+    ```
+
+3. PV is in `Failed` state:
+    ```bash
+    oc get pv <pv_name> -n <namespace> -o yaml
+    ```
+    If the pv is in `Failed` state, there might be a storage issue. Check the
+    events of the pv for more information.
+
+## Mitigation
+
+1. If resources are exhausted and pods are failing, additional resources need to
+be allocated to the storage cluster:
+    ```bash
+    oc patch -n openshift-storage storagecluster ocs-storagecluster \
+    --type merge \
+    --patch '{"spec": {"resourceProfile"="performance"}}'
+    ```
+
+2. Check upon the Ceph Storage: In case of issues with new OSDs, restart them
+and try again.
+
+3. Alternatively, one could try to restart the rook operator:
+    ```bash
+    oc rollout restart deployment rook-ceph-operator -n <namespace>
+    ```
+
+    After restarting, look for errors or warnings in the logs of rook-ceph-operator:
+    ```bash
+    oc logs -l app=rook-ceph-operator -n <namespace>
+    ```
+    Monitor the status of the pods and cephcluster post restart.
+
+4. If the PV is in `Pending` state, try to look at the errors in the events section:
path: alerts/openshift-container-storage-operator/StorageAutoScalingFailed.md
user: login: travisn
body: This seems like it should be in the diagnosis section, seems like it's mostly already covered by bullet 3 in the that section.
created_at: 2025-03-24T18:10:44Z
author_association: NONE
diff_hunk: @@ -0,0 +1,84 @@
+# StorageAutoScalingFailed
+
+## Meaning
+
+During automatic storage scaling, failures may occur due to various factors. In
+such instances, the `StorageAutoScalingFailed` alert is fired.
+
+## Impact
+
+The storage cannot be scaled at the moment.
+
+## Diagnosis
+
+There are multiple stages where the storage configurations might not have matched
+the requested storage parameters.
+
+1. OSD pod is in `Pending` state:
+    If it is in `Pending` state with the following logs, there is a high chance
+    it is due to the exhaustion of the resources.
+    ```bash
+    Warning  FailedScheduling  3m53s  default-scheduler  0/6 nodes are available: 1 Insufficient cpu, 2 node(s) didn't match Pod's node affinity/selector, 3 node(s) had untolerated taint {node-role.kubernetes.io/master: }. preemption: 0/6 nodes are available: 1 Insufficient cpu, 5 Preemption is not helpful for scheduling.
+    ```
+
+    Describe the pod in `Pending` state and check for the events.
+    ```bash
+    oc describe pod <osd_pod> -n <namespace>
+    ```
+
+    If there are no errors, retrieve the pvc of the pod in `Pending` state.
+    ```bash
+    pvcName=$(oc get pod <osd_pod> -n <namespace> -o jsonpath='{.spec.volumes[*].persistentVolumeClaim.claimName}')
+    oc get pvc $pvcName -n <namespace> -o yaml
+    ```
+    Check for the health and events of the PVC for detailed information.
+
+2. PVC is in `Pending` state:
+
+    ```bash
+    oc get pvc <pvc_name> -n <namespace> -o yaml
+    ```
+
+    Inspect the events of the pvc. Additionally, retrieve the pv of the pvc if
+    it is still in `pending` state.
+    ```bash
+    oc get pvc <pvc_name> -n <namespace> -o jsonpath='{.spec.volumeName}'
+    ```
+
+3. PV is in `Failed` state:
+    ```bash
+    oc get pv <pv_name> -n <namespace> -o yaml
+    ```
+    If the pv is in `Failed` state, there might be a storage issue. Check the
+    events of the pv for more information.
+
+## Mitigation
+
+1. If resources are exhausted and pods are failing, additional resources need to
+be allocated to the storage cluster:
+    ```bash
+    oc patch -n openshift-storage storagecluster ocs-storagecluster \
+    --type merge \
+    --patch '{"spec": {"resourceProfile"="performance"}}'
+    ```
+
+2. Check upon the Ceph Storage: In case of issues with new OSDs, restart them
+and try again.
+
+3. Alternatively, one could try to restart the rook operator:
path: alerts/openshift-container-storage-operator/StorageAutoScalingFailed.md
user: login: parth-gr
body: https://github.com/openshift/runbooks/pull/243#discussion_r2010152970
created_at: 2025-03-25T13:58:52Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,42 @@
+# StorageAutoScalingCapacityReached
+
+## Meaning
+
+The Storage Capacity Limit is the maximum storage that can be provisioned for
+the system. `StorageAutoScalingCapacityReached` alert is fired when the complete
+storage space has been utilized and hence additional storage cannot be allocated
+for the device class in the particular storage cluster.
+
+## Impact
+
+The storage cannot be auto-scaled at the moment.
+
+## Diagnosis
+
+Retrieve status of `storageCapacityLimitReached` from the StorageAutoScaler CR.
+The `resource_name` and `resource_namespace` would be specified in the alert itself.
path: alerts/openshift-container-storage-operator/StorageAutoScalingCapacityReached.md
user: login: travisn
body: Won't the namespace always be `openshift-storage`? If it's constant, maybe we don't need a variable for it.
created_at: 2025-03-25T19:50:43Z
author_association: NONE
diff_hunk: @@ -0,0 +1,42 @@
+# StorageAutoScalingCapacityReached
+
+## Meaning
+
+The Storage Capacity Limit is the maximum storage that can be provisioned for
+the system. `StorageAutoScalingCapacityReached` alert is fired when the complete
+storage space has been utilized and hence additional storage cannot be allocated
+for the device class in the particular storage cluster.
+
+## Impact
+
+The storage cannot be auto-scaled at the moment.
+
+## Diagnosis
+
+Retrieve status of `storageCapacityLimitReached` from the StorageAutoScaler CR.
+The `resource_name` and `resource_namespace` would be specified in the alert itself.
+
+```bash
+oc get storageautoscaler <resource_name> -n <resource_namespace> -o json | jq -r '.status.storageCapacityLimitReached'
+```
+
+If status of `storageCapacityLimitReached` is `true` then maximum capacity of
+the storage has been utilized.
+
+## Mitigation
+
+- Scale storage: Execute the following command with the updated `storageCapacityLimit`-
+  ```bash
+  oc patch storageautoscaler <resource_name> -n <resource_namespace> --type=merge --patch '{"spec":{"storageCapacityLimit":<storageCapacityLimit>}}'
+  ```
+  Note: The `status.error.message` in the StorageAutoScaler CR will show the
+  minimum required increase for `storageCapacityLimit`.Ensure that the specified
+  capacity is atleast that value or higher.
path: alerts/openshift-container-storage-operator/StorageAutoScalingCapacityReached.md
user: login: travisn
body: ```suggestion
  capacity is at least that value or higher.
```
created_at: 2025-03-25T19:51:29Z
author_association: NONE
diff_hunk: @@ -0,0 +1,42 @@
+# StorageAutoScalingCapacityReached
+
+## Meaning
+
+The Storage Capacity Limit is the maximum storage that can be provisioned for
+the system. `StorageAutoScalingCapacityReached` alert is fired when the complete
+storage space has been utilized and hence additional storage cannot be allocated
+for the device class in the particular storage cluster.
+
+## Impact
+
+The storage cannot be auto-scaled at the moment.
+
+## Diagnosis
+
+Retrieve status of `storageCapacityLimitReached` from the StorageAutoScaler CR.
+The `resource_name` and `resource_namespace` would be specified in the alert itself.
+
+```bash
+oc get storageautoscaler <resource_name> -n <resource_namespace> -o json | jq -r '.status.storageCapacityLimitReached'
+```
+
+If status of `storageCapacityLimitReached` is `true` then maximum capacity of
+the storage has been utilized.
+
+## Mitigation
+
path: alerts/openshift-container-storage-operator/StorageAutoScalingCapacityReached.md
user: login: travisn
body: Need a sentence to describe that they can choose between two possible fixes? Otherwise, they might think they need both steps.
created_at: 2025-03-25T19:52:22Z
author_association: NONE
diff_hunk: @@ -0,0 +1,42 @@
+# StorageAutoScalingCapacityReached
+
+## Meaning
+
+The Storage Capacity Limit is the maximum storage that can be provisioned for
+the system. `StorageAutoScalingCapacityReached` alert is fired when the complete
+storage space has been utilized and hence additional storage cannot be allocated
+for the device class in the particular storage cluster.
+
+## Impact
+
+The storage cannot be auto-scaled at the moment.
+
+## Diagnosis
+
+Retrieve status of `storageCapacityLimitReached` from the StorageAutoScaler CR.
+The `resource_name` and `resource_namespace` would be specified in the alert itself.
+
+```bash
+oc get storageautoscaler <resource_name> -n <resource_namespace> -o json | jq -r '.status.storageCapacityLimitReached'
+```
+
+If status of `storageCapacityLimitReached` is `true` then maximum capacity of
+the storage has been utilized.
+
+## Mitigation
+
+- Scale storage: Execute the following command with the updated `storageCapacityLimit`-
+  ```bash
+  oc patch storageautoscaler <resource_name> -n <resource_namespace> --type=merge --patch '{"spec":{"storageCapacityLimit":<storageCapacityLimit>}}'
+  ```
+  Note: The `status.error.message` in the StorageAutoScaler CR will show the
+  minimum required increase for `storageCapacityLimit`.Ensure that the specified
+  capacity is atleast that value or higher.
+
+- Disable Auto-Scale and scale directly: Additional resources, such as OSDs, can
path: alerts/openshift-container-storage-operator/StorageAutoScalingCapacityReached.md
user: login: travisn
body: ```suggestion
- If auto scaling is no longer desired, disable the feature. Additional storage can instead be added when needed via the ODF console. 
```
created_at: 2025-03-25T19:55:03Z
author_association: NONE
diff_hunk: @@ -0,0 +1,75 @@
+# StorageAutoScalingFailed
+
+## Meaning
+
+During automatic storage scaling, failures may occur due to various factors. In
+such instances, the `StorageAutoScalingFailed` alert is fired.
+
+## Impact
+
+The storage cannot be scaled at the moment.
+
+## Diagnosis
+
+There are multiple stages where the storage configurations might not have matched
+the requested storage parameters.
+
+1. OSD pod is in `Pending` state:
+    If it is in `Pending` state with the following logs, there is a high chance
+    it is due to the exhaustion of the resources.
+    ```bash
+    Warning  FailedScheduling  3m53s  default-scheduler  0/6 nodes are available: 1 Insufficient cpu, 2 node(s) didn't match Pod's node affinity/selector, 3 node(s) had untolerated taint {node-role.kubernetes.io/master: }. preemption: 0/6 nodes are available: 1 Insufficient cpu, 5 Preemption is not helpful for scheduling.
+    ```
+
+    Describe the pod in `Pending` state and check for the events.
+    ```bash
+    oc describe pod <osd_pod> -n <namespace>
+    ```
+
+    If there are no errors, retrieve the pvc of the pod in `Pending` state.
path: alerts/openshift-container-storage-operator/StorageAutoScalingFailed.md
user: login: travisn
body: Are lines 29-34 already covered in the next section? It seems duplicative here.
created_at: 2025-03-25T19:57:30Z
author_association: NONE
diff_hunk: @@ -0,0 +1,75 @@
+# StorageAutoScalingFailed
+
+## Meaning
+
+During automatic storage scaling, failures may occur due to various factors. In
+such instances, the `StorageAutoScalingFailed` alert is fired.
+
+## Impact
+
+The storage cannot be scaled at the moment.
+
+## Diagnosis
+
+There are multiple stages where the storage configurations might not have matched
+the requested storage parameters.
+
+1. OSD pod is in `Pending` state:
+    If it is in `Pending` state with the following logs, there is a high chance
+    it is due to the exhaustion of the resources.
+    ```bash
+    Warning  FailedScheduling  3m53s  default-scheduler  0/6 nodes are available: 1 Insufficient cpu, 2 node(s) didn't match Pod's node affinity/selector, 3 node(s) had untolerated taint {node-role.kubernetes.io/master: }. preemption: 0/6 nodes are available: 1 Insufficient cpu, 5 Preemption is not helpful for scheduling.
+    ```
+
+    Describe the pod in `Pending` state and check for the events.
+    ```bash
+    oc describe pod <osd_pod> -n <namespace>
+    ```
+
+    If there are no errors, retrieve the pvc of the pod in `Pending` state.
+    ```bash
+    pvcName=$(oc get pod <osd_pod> -n <namespace> -o jsonpath='{.spec.volumes[*].persistentVolumeClaim.claimName}')
+    oc get pvc $pvcName -n <namespace> -o yaml
+    ```
+    Check for the health and events of the PVC for detailed information.
+
+2. PVC is in `Pending` state:
path: alerts/openshift-container-storage-operator/StorageAutoScalingFailed.md
user: login: travisn
body: @parth-gr In your testing have you seen the PVC in pending state? Commonly I would only think this is seen with LSO when there might not be more devices available for the horizontal scale. 
created_at: 2025-03-25T19:58:49Z
author_association: NONE
diff_hunk: @@ -0,0 +1,75 @@
+# StorageAutoScalingFailed
+
+## Meaning
+
+During automatic storage scaling, failures may occur due to various factors. In
+such instances, the `StorageAutoScalingFailed` alert is fired.
+
+## Impact
+
+The storage cannot be scaled at the moment.
+
+## Diagnosis
+
+There are multiple stages where the storage configurations might not have matched
+the requested storage parameters.
+
+1. OSD pod is in `Pending` state:
+    If it is in `Pending` state with the following logs, there is a high chance
+    it is due to the exhaustion of the resources.
+    ```bash
+    Warning  FailedScheduling  3m53s  default-scheduler  0/6 nodes are available: 1 Insufficient cpu, 2 node(s) didn't match Pod's node affinity/selector, 3 node(s) had untolerated taint {node-role.kubernetes.io/master: }. preemption: 0/6 nodes are available: 1 Insufficient cpu, 5 Preemption is not helpful for scheduling.
+    ```
+
+    Describe the pod in `Pending` state and check for the events.
+    ```bash
+    oc describe pod <osd_pod> -n <namespace>
+    ```
+
+    If there are no errors, retrieve the pvc of the pod in `Pending` state.
+    ```bash
+    pvcName=$(oc get pod <osd_pod> -n <namespace> -o jsonpath='{.spec.volumes[*].persistentVolumeClaim.claimName}')
+    oc get pvc $pvcName -n <namespace> -o yaml
+    ```
+    Check for the health and events of the PVC for detailed information.
+
+2. PVC is in `Pending` state:
+
+    ```bash
+    oc get pvc <pvc_name> -n <namespace> -o yaml
+    ```
+
+    Inspect the events of the pvc. Additionally, retrieve the pv of the pvc if
+    it is still in `pending` state.
+    ```bash
+    oc get pvc <pvc_name> -n <namespace> -o jsonpath='{.spec.volumeName}'
+    ```
+
+3. PV is in `Failed` state:
path: alerts/openshift-container-storage-operator/StorageAutoScalingFailed.md
user: login: travisn
body: @parth-gr When would this happen? For vertical scaling I don't expect the PVs to be in failed state, and I don't know why they would be failed for horizontal scale.
created_at: 2025-03-25T19:59:38Z
author_association: NONE
diff_hunk: @@ -0,0 +1,75 @@
+# StorageAutoScalingFailed
+
+## Meaning
+
+During automatic storage scaling, failures may occur due to various factors. In
+such instances, the `StorageAutoScalingFailed` alert is fired.
+
+## Impact
+
+The storage cannot be scaled at the moment.
+
+## Diagnosis
+
+There are multiple stages where the storage configurations might not have matched
+the requested storage parameters.
+
+1. OSD pod is in `Pending` state:
+    If it is in `Pending` state with the following logs, there is a high chance
+    it is due to the exhaustion of the resources.
+    ```bash
+    Warning  FailedScheduling  3m53s  default-scheduler  0/6 nodes are available: 1 Insufficient cpu, 2 node(s) didn't match Pod's node affinity/selector, 3 node(s) had untolerated taint {node-role.kubernetes.io/master: }. preemption: 0/6 nodes are available: 1 Insufficient cpu, 5 Preemption is not helpful for scheduling.
+    ```
+
+    Describe the pod in `Pending` state and check for the events.
+    ```bash
+    oc describe pod <osd_pod> -n <namespace>
+    ```
+
+    If there are no errors, retrieve the pvc of the pod in `Pending` state.
+    ```bash
+    pvcName=$(oc get pod <osd_pod> -n <namespace> -o jsonpath='{.spec.volumes[*].persistentVolumeClaim.claimName}')
+    oc get pvc $pvcName -n <namespace> -o yaml
+    ```
+    Check for the health and events of the PVC for detailed information.
+
+2. PVC is in `Pending` state:
+
+    ```bash
+    oc get pvc <pvc_name> -n <namespace> -o yaml
+    ```
+
+    Inspect the events of the pvc. Additionally, retrieve the pv of the pvc if
+    it is still in `pending` state.
+    ```bash
+    oc get pvc <pvc_name> -n <namespace> -o jsonpath='{.spec.volumeName}'
+    ```
+
+3. PV is in `Failed` state:
+    ```bash
+    oc get pv <pv_name> -n <namespace> -o yaml
+    ```
+    If the pv is in `Failed` state, there might be a storage issue. Check the
+    events of the pv for more information.
+
+## Mitigation
+
+1. If resources are depleted and pods start failing, allocate additional
+resources to the storage cluster. Possible solutions include adding more nodes
+or scaling up CPU and memory on existing nodes.
+- For instructions on adding nodes, please refer to this document: [Adding a node](https://docs.redhat.com/en/documentation/red_hat_openshift_data_foundation/4.17/html-single/scaling_storage/index#adding_a_node)
+
+2. Check upon the Ceph Storage: In case the OSDs are `down` or `out`, restart them
+and try again.
+
+3. During vertical scaling, if the pv takes time to resize, then the osds would not
path: alerts/openshift-container-storage-operator/StorageAutoScalingFailed.md
user: login: travisn
body: @parth-gr I don't think restarting the rook operator will help. Restarting the OSD pods sometimes can help nudge the resize to completion though. 
created_at: 2025-03-25T20:00:53Z
author_association: NONE
diff_hunk: @@ -0,0 +1,75 @@
+# StorageAutoScalingFailed
+
+## Meaning
+
+During automatic storage scaling, failures may occur due to various factors. In
+such instances, the `StorageAutoScalingFailed` alert is fired.
+
+## Impact
+
+The storage cannot be scaled at the moment.
+
+## Diagnosis
+
+There are multiple stages where the storage configurations might not have matched
+the requested storage parameters.
+
+1. OSD pod is in `Pending` state:
+    If it is in `Pending` state with the following logs, there is a high chance
+    it is due to the exhaustion of the resources.
+    ```bash
+    Warning  FailedScheduling  3m53s  default-scheduler  0/6 nodes are available: 1 Insufficient cpu, 2 node(s) didn't match Pod's node affinity/selector, 3 node(s) had untolerated taint {node-role.kubernetes.io/master: }. preemption: 0/6 nodes are available: 1 Insufficient cpu, 5 Preemption is not helpful for scheduling.
+    ```
+
+    Describe the pod in `Pending` state and check for the events.
+    ```bash
+    oc describe pod <osd_pod> -n <namespace>
+    ```
+
+    If there are no errors, retrieve the pvc of the pod in `Pending` state.
+    ```bash
+    pvcName=$(oc get pod <osd_pod> -n <namespace> -o jsonpath='{.spec.volumes[*].persistentVolumeClaim.claimName}')
+    oc get pvc $pvcName -n <namespace> -o yaml
+    ```
+    Check for the health and events of the PVC for detailed information.
+
+2. PVC is in `Pending` state:
+
+    ```bash
+    oc get pvc <pvc_name> -n <namespace> -o yaml
+    ```
+
+    Inspect the events of the pvc. Additionally, retrieve the pv of the pvc if
+    it is still in `pending` state.
+    ```bash
+    oc get pvc <pvc_name> -n <namespace> -o jsonpath='{.spec.volumeName}'
+    ```
+
+3. PV is in `Failed` state:
+    ```bash
+    oc get pv <pv_name> -n <namespace> -o yaml
+    ```
+    If the pv is in `Failed` state, there might be a storage issue. Check the
+    events of the pv for more information.
+
+## Mitigation
+
+1. If resources are depleted and pods start failing, allocate additional
+resources to the storage cluster. Possible solutions include adding more nodes
+or scaling up CPU and memory on existing nodes.
+- For instructions on adding nodes, please refer to this document: [Adding a node](https://docs.redhat.com/en/documentation/red_hat_openshift_data_foundation/4.17/html-single/scaling_storage/index#adding_a_node)
path: alerts/openshift-container-storage-operator/StorageAutoScalingFailed.md
user: login: travisn
body: Is there a `latest` link? Or will the 4.17 version be updated with each release?
created_at: 2025-03-25T20:01:54Z
author_association: NONE
diff_hunk: @@ -0,0 +1,75 @@
+# StorageAutoScalingFailed
+
+## Meaning
+
+During automatic storage scaling, failures may occur due to various factors. In
+such instances, the `StorageAutoScalingFailed` alert is fired.
+
+## Impact
+
+The storage cannot be scaled at the moment.
+
+## Diagnosis
+
+There are multiple stages where the storage configurations might not have matched
+the requested storage parameters.
+
+1. OSD pod is in `Pending` state:
+    If it is in `Pending` state with the following logs, there is a high chance
+    it is due to the exhaustion of the resources.
+    ```bash
+    Warning  FailedScheduling  3m53s  default-scheduler  0/6 nodes are available: 1 Insufficient cpu, 2 node(s) didn't match Pod's node affinity/selector, 3 node(s) had untolerated taint {node-role.kubernetes.io/master: }. preemption: 0/6 nodes are available: 1 Insufficient cpu, 5 Preemption is not helpful for scheduling.
+    ```
+
+    Describe the pod in `Pending` state and check for the events.
+    ```bash
+    oc describe pod <osd_pod> -n <namespace>
+    ```
+
+    If there are no errors, retrieve the pvc of the pod in `Pending` state.
path: alerts/openshift-container-storage-operator/StorageAutoScalingFailed.md
user: login: ShravaniVangur
body: The lines 29-34 specify how the pvc name can be extracted from the osd pod. The next section is a general way of describing a pvc. That is why it has been split into two different sections. 
created_at: 2025-03-26T05:23:32Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,42 @@
+# StorageAutoScalingCapacityReached
+
+## Meaning
+
+The Storage Capacity Limit is the maximum storage that can be provisioned for
+the system. `StorageAutoScalingCapacityReached` alert is fired when the complete
+storage space has been utilized and hence additional storage cannot be allocated
+for the device class in the particular storage cluster.
+
+## Impact
+
+The storage cannot be auto-scaled at the moment.
+
+## Diagnosis
+
+Retrieve status of `storageCapacityLimitReached` from the StorageAutoScaler CR.
+The `resource_name` and `resource_namespace` would be specified in the alert itself.
path: alerts/openshift-container-storage-operator/StorageAutoScalingCapacityReached.md
user: login: parth-gr
body: It won't, in rosa we deploy at `odf-storage` namespace
created_at: 2025-03-26T06:11:12Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,75 @@
+# StorageAutoScalingFailed
+
+## Meaning
+
+During automatic storage scaling, failures may occur due to various factors. In
+such instances, the `StorageAutoScalingFailed` alert is fired.
+
+## Impact
+
+The storage cannot be scaled at the moment.
+
+## Diagnosis
+
+There are multiple stages where the storage configurations might not have matched
+the requested storage parameters.
+
+1. OSD pod is in `Pending` state:
+    If it is in `Pending` state with the following logs, there is a high chance
+    it is due to the exhaustion of the resources.
+    ```bash
+    Warning  FailedScheduling  3m53s  default-scheduler  0/6 nodes are available: 1 Insufficient cpu, 2 node(s) didn't match Pod's node affinity/selector, 3 node(s) had untolerated taint {node-role.kubernetes.io/master: }. preemption: 0/6 nodes are available: 1 Insufficient cpu, 5 Preemption is not helpful for scheduling.
+    ```
+
+    Describe the pod in `Pending` state and check for the events.
+    ```bash
+    oc describe pod <osd_pod> -n <namespace>
+    ```
+
+    If there are no errors, retrieve the pvc of the pod in `Pending` state.
+    ```bash
+    pvcName=$(oc get pod <osd_pod> -n <namespace> -o jsonpath='{.spec.volumes[*].persistentVolumeClaim.claimName}')
+    oc get pvc $pvcName -n <namespace> -o yaml
+    ```
+    Check for the health and events of the PVC for detailed information.
+
+2. PVC is in `Pending` state:
path: alerts/openshift-container-storage-operator/StorageAutoScalingFailed.md
user: login: parth-gr
body: Not seen that usual, but can happen if csi driver is not responding to the provision request
created_at: 2025-03-26T06:12:26Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,75 @@
+# StorageAutoScalingFailed
+
+## Meaning
+
+During automatic storage scaling, failures may occur due to various factors. In
+such instances, the `StorageAutoScalingFailed` alert is fired.
+
+## Impact
+
+The storage cannot be scaled at the moment.
+
+## Diagnosis
+
+There are multiple stages where the storage configurations might not have matched
+the requested storage parameters.
+
+1. OSD pod is in `Pending` state:
+    If it is in `Pending` state with the following logs, there is a high chance
+    it is due to the exhaustion of the resources.
+    ```bash
+    Warning  FailedScheduling  3m53s  default-scheduler  0/6 nodes are available: 1 Insufficient cpu, 2 node(s) didn't match Pod's node affinity/selector, 3 node(s) had untolerated taint {node-role.kubernetes.io/master: }. preemption: 0/6 nodes are available: 1 Insufficient cpu, 5 Preemption is not helpful for scheduling.
+    ```
+
+    Describe the pod in `Pending` state and check for the events.
+    ```bash
+    oc describe pod <osd_pod> -n <namespace>
+    ```
+
+    If there are no errors, retrieve the pvc of the pod in `Pending` state.
+    ```bash
+    pvcName=$(oc get pod <osd_pod> -n <namespace> -o jsonpath='{.spec.volumes[*].persistentVolumeClaim.claimName}')
+    oc get pvc $pvcName -n <namespace> -o yaml
+    ```
+    Check for the health and events of the PVC for detailed information.
+
+2. PVC is in `Pending` state:
+
+    ```bash
+    oc get pvc <pvc_name> -n <namespace> -o yaml
+    ```
+
+    Inspect the events of the pvc. Additionally, retrieve the pv of the pvc if
+    it is still in `pending` state.
+    ```bash
+    oc get pvc <pvc_name> -n <namespace> -o jsonpath='{.spec.volumeName}'
+    ```
+
+3. PV is in `Failed` state:
path: alerts/openshift-container-storage-operator/StorageAutoScalingFailed.md
user: login: parth-gr
body: Pv's should be pending state, @ShravaniVangur 
created_at: 2025-03-26T06:13:05Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,75 @@
+# StorageAutoScalingFailed
+
+## Meaning
+
+During automatic storage scaling, failures may occur due to various factors. In
+such instances, the `StorageAutoScalingFailed` alert is fired.
+
+## Impact
+
+The storage cannot be scaled at the moment.
+
+## Diagnosis
+
+There are multiple stages where the storage configurations might not have matched
+the requested storage parameters.
+
+1. OSD pod is in `Pending` state:
+    If it is in `Pending` state with the following logs, there is a high chance
+    it is due to the exhaustion of the resources.
+    ```bash
+    Warning  FailedScheduling  3m53s  default-scheduler  0/6 nodes are available: 1 Insufficient cpu, 2 node(s) didn't match Pod's node affinity/selector, 3 node(s) had untolerated taint {node-role.kubernetes.io/master: }. preemption: 0/6 nodes are available: 1 Insufficient cpu, 5 Preemption is not helpful for scheduling.
+    ```
+
+    Describe the pod in `Pending` state and check for the events.
+    ```bash
+    oc describe pod <osd_pod> -n <namespace>
+    ```
+
+    If there are no errors, retrieve the pvc of the pod in `Pending` state.
+    ```bash
+    pvcName=$(oc get pod <osd_pod> -n <namespace> -o jsonpath='{.spec.volumes[*].persistentVolumeClaim.claimName}')
+    oc get pvc $pvcName -n <namespace> -o yaml
+    ```
+    Check for the health and events of the PVC for detailed information.
+
+2. PVC is in `Pending` state:
+
+    ```bash
+    oc get pvc <pvc_name> -n <namespace> -o yaml
+    ```
+
+    Inspect the events of the pvc. Additionally, retrieve the pv of the pvc if
+    it is still in `pending` state.
+    ```bash
+    oc get pvc <pvc_name> -n <namespace> -o jsonpath='{.spec.volumeName}'
+    ```
+
+3. PV is in `Failed` state:
+    ```bash
+    oc get pv <pv_name> -n <namespace> -o yaml
+    ```
+    If the pv is in `Failed` state, there might be a storage issue. Check the
+    events of the pv for more information.
+
+## Mitigation
+
+1. If resources are depleted and pods start failing, allocate additional
+resources to the storage cluster. Possible solutions include adding more nodes
+or scaling up CPU and memory on existing nodes.
+- For instructions on adding nodes, please refer to this document: [Adding a node](https://docs.redhat.com/en/documentation/red_hat_openshift_data_foundation/4.17/html-single/scaling_storage/index#adding_a_node)
+
+2. Check upon the Ceph Storage: In case the OSDs are `down` or `out`, restart them
+and try again.
+
+3. During vertical scaling, if the pv takes time to resize, then the osds would not
path: alerts/openshift-container-storage-operator/StorageAutoScalingFailed.md
user: login: parth-gr
body: Restarting the operator helps in the vertical scaling, When we need to update the crush-weight, which is not updated by `bluefs-bdev-expand` cmd and rook operator separately does that during osd upgrades.

that's correct it doesn't help in osd to get resize, but it helps in correcting the crush-weight if osd vertical scaling was manually triggered by re-starting the osd pod

@ShravaniVangur can you frame in that way, if we are doing osd pod restart better to also restart operator after that 
created_at: 2025-03-26T06:17:31Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,75 @@
+# StorageAutoScalingFailed
+
+## Meaning
+
+During automatic storage scaling, failures may occur due to various factors. In
+such instances, the `StorageAutoScalingFailed` alert is fired.
+
+## Impact
+
+The storage cannot be scaled at the moment.
+
+## Diagnosis
+
+There are multiple stages where the storage configurations might not have matched
+the requested storage parameters.
+
+1. OSD pod is in `Pending` state:
+    If it is in `Pending` state with the following logs, there is a high chance
+    it is due to the exhaustion of the resources.
+    ```bash
+    Warning  FailedScheduling  3m53s  default-scheduler  0/6 nodes are available: 1 Insufficient cpu, 2 node(s) didn't match Pod's node affinity/selector, 3 node(s) had untolerated taint {node-role.kubernetes.io/master: }. preemption: 0/6 nodes are available: 1 Insufficient cpu, 5 Preemption is not helpful for scheduling.
+    ```
+
+    Describe the pod in `Pending` state and check for the events.
+    ```bash
+    oc describe pod <osd_pod> -n <namespace>
+    ```
+
+    If there are no errors, retrieve the pvc of the pod in `Pending` state.
+    ```bash
+    pvcName=$(oc get pod <osd_pod> -n <namespace> -o jsonpath='{.spec.volumes[*].persistentVolumeClaim.claimName}')
+    oc get pvc $pvcName -n <namespace> -o yaml
+    ```
+    Check for the health and events of the PVC for detailed information.
+
+2. PVC is in `Pending` state:
+
+    ```bash
+    oc get pvc <pvc_name> -n <namespace> -o yaml
+    ```
+
+    Inspect the events of the pvc. Additionally, retrieve the pv of the pvc if
+    it is still in `pending` state.
+    ```bash
+    oc get pvc <pvc_name> -n <namespace> -o jsonpath='{.spec.volumeName}'
+    ```
+
+3. PV is in `Failed` state:
+    ```bash
+    oc get pv <pv_name> -n <namespace> -o yaml
+    ```
+    If the pv is in `Failed` state, there might be a storage issue. Check the
+    events of the pv for more information.
+
+## Mitigation
+
+1. If resources are depleted and pods start failing, allocate additional
+resources to the storage cluster. Possible solutions include adding more nodes
+or scaling up CPU and memory on existing nodes.
+- For instructions on adding nodes, please refer to this document: [Adding a node](https://docs.redhat.com/en/documentation/red_hat_openshift_data_foundation/4.17/html-single/scaling_storage/index#adding_a_node)
path: alerts/openshift-container-storage-operator/StorageAutoScalingFailed.md
user: login: parth-gr
body: @ShravaniVangur can you check with support team on this, otherwise we have to maintain it with every release

Or can we add <odf-version> and provide a 4.17 example
created_at: 2025-03-26T06:18:30Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,75 @@
+# StorageAutoScalingFailed
+
+## Meaning
+
+During automatic storage scaling, failures may occur due to various factors. In
+such instances, the `StorageAutoScalingFailed` alert is fired.
+
+## Impact
+
+The storage cannot be scaled at the moment.
+
+## Diagnosis
+
+There are multiple stages where the storage configurations might not have matched
+the requested storage parameters.
+
+1. OSD pod is in `Pending` state:
+    If it is in `Pending` state with the following logs, there is a high chance
+    it is due to the exhaustion of the resources.
+    ```bash
+    Warning  FailedScheduling  3m53s  default-scheduler  0/6 nodes are available: 1 Insufficient cpu, 2 node(s) didn't match Pod's node affinity/selector, 3 node(s) had untolerated taint {node-role.kubernetes.io/master: }. preemption: 0/6 nodes are available: 1 Insufficient cpu, 5 Preemption is not helpful for scheduling.
+    ```
+
+    Describe the pod in `Pending` state and check for the events.
+    ```bash
+    oc describe pod <osd_pod> -n <namespace>
+    ```
+
+    If there are no errors, retrieve the pvc of the pod in `Pending` state.
+    ```bash
+    pvcName=$(oc get pod <osd_pod> -n <namespace> -o jsonpath='{.spec.volumes[*].persistentVolumeClaim.claimName}')
+    oc get pvc $pvcName -n <namespace> -o yaml
+    ```
+    Check for the health and events of the PVC for detailed information.
+
+2. PVC is in `Pending` state:
+
+    ```bash
+    oc get pvc <pvc_name> -n <namespace> -o yaml
+    ```
+
+    Inspect the events of the pvc. Additionally, retrieve the pv of the pvc if
+    it is still in `pending` state.
+    ```bash
+    oc get pvc <pvc_name> -n <namespace> -o jsonpath='{.spec.volumeName}'
+    ```
+
+3. PV is in `Failed` state:
+    ```bash
+    oc get pv <pv_name> -n <namespace> -o yaml
+    ```
+    If the pv is in `Failed` state, there might be a storage issue. Check the
+    events of the pv for more information.
+
+## Mitigation
+
+1. If resources are depleted and pods start failing, allocate additional
+resources to the storage cluster. Possible solutions include adding more nodes
+or scaling up CPU and memory on existing nodes.
+- For instructions on adding nodes, please refer to this document: [Adding a node](https://docs.redhat.com/en/documentation/red_hat_openshift_data_foundation/4.18/html-single/scaling_storage/index#adding_a_node)
path: alerts/openshift-container-storage-operator/StorageAutoScalingFailed.md
user: login: parth-gr
body: change the <odf-version> with the desired odf version you are using,

https://docs.redhat.com/en/documentation/red_hat_openshift_data_foundation/"odf-version"/html-single/scaling_storage/index#adding_a_node

example: 
odf-version = 4.18
(https://docs.redhat.com/en/documentation/red_hat_openshift_data_foundation/4.18/html-single/scaling_storage/index#adding_a_node)
created_at: 2025-03-26T07:55:07Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,75 @@
+# StorageAutoScalingFailed
+
+## Meaning
+
+During automatic storage scaling, failures may occur due to various factors. In
+such instances, the `StorageAutoScalingFailed` alert is fired.
+
+## Impact
+
+The storage cannot be scaled at the moment.
+
+## Diagnosis
+
+There are multiple stages where the storage configurations might not have matched
+the requested storage parameters.
+
+1. OSD pod is in `Pending` state:
+    If it is in `Pending` state with the following logs, there is a high chance
+    it is due to the exhaustion of the resources.
+    ```bash
+    Warning  FailedScheduling  3m53s  default-scheduler  0/6 nodes are available: 1 Insufficient cpu, 2 node(s) didn't match Pod's node affinity/selector, 3 node(s) had untolerated taint {node-role.kubernetes.io/master: }. preemption: 0/6 nodes are available: 1 Insufficient cpu, 5 Preemption is not helpful for scheduling.
+    ```
+
+    Describe the pod in `Pending` state and check for the events.
+    ```bash
+    oc describe pod <osd_pod> -n <namespace>
+    ```
+
+    If there are no errors, retrieve the pvc of the pod in `Pending` state.
+    ```bash
+    pvcName=$(oc get pod <osd_pod> -n <namespace> -o jsonpath='{.spec.volumes[*].persistentVolumeClaim.claimName}')
+    oc get pvc $pvcName -n <namespace> -o yaml
+    ```
+    Check for the health and events of the PVC for detailed information.
+
+2. PVC is in `Pending` state:
+
+    ```bash
+    oc get pvc <pvc_name> -n <namespace> -o yaml
+    ```
+
+    Inspect the events of the pvc. Additionally, retrieve the pv of the pvc if
+    it is still in `pending` state.
+    ```bash
+    oc get pvc <pvc_name> -n <namespace> -o jsonpath='{.spec.volumeName}'
+    ```
+
+3. PV is in `Failed` state:
+    ```bash
+    oc get pv <pv_name> -n <namespace> -o yaml
+    ```
+    If the pv is in `Failed` state, there might be a storage issue. Check the
+    events of the pv for more information.
+
+## Mitigation
+
+1. If resources are depleted and pods start failing, allocate additional
+resources to the storage cluster. Possible solutions include adding more nodes
+or scaling up CPU and memory on existing nodes.
+- For instructions on adding nodes, please refer to this document: [Adding a node](https://docs.redhat.com/en/documentation/red_hat_openshift_data_foundation/4.18/html-single/scaling_storage/index#adding_a_node)
path: alerts/openshift-container-storage-operator/StorageAutoScalingFailed.md
user: login: ShravaniVangur
body: Currently 4.18 link has been added which will have the label - `Adding a node`. Does it have to be changed to show the complete link and quote the example or does this work fine? 
created_at: 2025-03-26T09:13:17Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,75 @@
+# StorageAutoScalingFailed
+
+## Meaning
+
+During automatic storage scaling, failures may occur due to various factors. In
+such instances, the `StorageAutoScalingFailed` alert is fired.
+
+## Impact
+
+The storage cannot be scaled at the moment.
+
+## Diagnosis
+
+There are multiple stages where the storage configurations might not have matched
+the requested storage parameters.
+
+1. OSD pod is in `Pending` state:
+    If it is in `Pending` state with the following logs, there is a high chance
+    it is due to the exhaustion of the resources.
+    ```bash
+    Warning  FailedScheduling  3m53s  default-scheduler  0/6 nodes are available: 1 Insufficient cpu, 2 node(s) didn't match Pod's node affinity/selector, 3 node(s) had untolerated taint {node-role.kubernetes.io/master: }. preemption: 0/6 nodes are available: 1 Insufficient cpu, 5 Preemption is not helpful for scheduling.
+    ```
+
+    Describe the pod in `Pending` state and check for the events.
+    ```bash
+    oc describe pod <osd_pod> -n <namespace>
+    ```
+
+    If there are no errors, retrieve the pvc of the pod in `Pending` state.
+    ```bash
+    pvcName=$(oc get pod <osd_pod> -n <namespace> -o jsonpath='{.spec.volumes[*].persistentVolumeClaim.claimName}')
+    oc get pvc $pvcName -n <namespace> -o yaml
+    ```
+    Check for the health and events of the PVC for detailed information.
+
+2. PVC is in `Pending` state:
+
+    ```bash
+    oc get pvc <pvc_name> -n <namespace> -o yaml
+    ```
+
+    Inspect the events of the pvc. Additionally, retrieve the pv of the pvc if
+    it is still in `pending` state.
+    ```bash
+    oc get pvc <pvc_name> -n <namespace> -o jsonpath='{.spec.volumeName}'
+    ```
+
+3. PV is in `Failed` state:
+    ```bash
+    oc get pv <pv_name> -n <namespace> -o yaml
+    ```
+    If the pv is in `Failed` state, there might be a storage issue. Check the
+    events of the pv for more information.
+
+## Mitigation
+
+1. If resources are depleted and pods start failing, allocate additional
+resources to the storage cluster. Possible solutions include adding more nodes
+or scaling up CPU and memory on existing nodes.
+- For instructions on adding nodes, please refer to this document: [Adding a node](https://docs.redhat.com/en/documentation/red_hat_openshift_data_foundation/4.18/html-single/scaling_storage/index#adding_a_node)
path: alerts/openshift-container-storage-operator/StorageAutoScalingFailed.md
user: login: parth-gr
body: if you just add 4.18 link then you have to maintain it in every release to update it,
So it would be good if you can provide 4.18 link in example and use a <odf-version> tag in the link.
created_at: 2025-03-26T11:38:54Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,76 @@
+# StorageAutoScalingFailed
+
+## Meaning
+
+During automatic storage scaling, failures may occur due to various factors. In
+such instances, the `StorageAutoScalingFailed` alert is fired.
+
+## Impact
+
+The storage cannot be scaled at the moment.
+
+## Diagnosis
+
+There are multiple stages where the storage configurations might not have matched
+the requested storage parameters.
+
+1. OSD pod is in `Pending` state:
+    If it is in `Pending` state with the following logs, there is a high chance
+    it is due to the exhaustion of the resources.
+    ```bash
+    Warning  FailedScheduling  3m53s  default-scheduler  0/6 nodes are available: 1 Insufficient cpu, 2 node(s) didn't match Pod's node affinity/selector, 3 node(s) had untolerated taint {node-role.kubernetes.io/master: }. preemption: 0/6 nodes are available: 1 Insufficient cpu, 5 Preemption is not helpful for scheduling.
path: alerts/openshift-container-storage-operator/StorageAutoScalingFailed.md
user: login: travisn
body: This warning is a result of describing the pod with the command below on line 26. Need to change the order so the output comes after the command?
created_at: 2025-03-26T14:01:55Z
author_association: NONE
diff_hunk: @@ -0,0 +1,75 @@
+# StorageAutoScalingFailed
+
+## Meaning
+
+During automatic storage scaling, failures may occur due to various factors. In
+such instances, the `StorageAutoScalingFailed` alert is fired.
+
+## Impact
+
+The storage cannot be scaled at the moment.
+
+## Diagnosis
+
+There are multiple stages where the storage configurations might not have matched
+the requested storage parameters.
+
+1. OSD pod is in `Pending` state:
+    If it is in `Pending` state with the following logs, there is a high chance
+    it is due to the exhaustion of the resources.
+    ```bash
+    Warning  FailedScheduling  3m53s  default-scheduler  0/6 nodes are available: 1 Insufficient cpu, 2 node(s) didn't match Pod's node affinity/selector, 3 node(s) had untolerated taint {node-role.kubernetes.io/master: }. preemption: 0/6 nodes are available: 1 Insufficient cpu, 5 Preemption is not helpful for scheduling.
+    ```
+
+    Describe the pod in `Pending` state and check for the events.
+    ```bash
+    oc describe pod <osd_pod> -n <namespace>
+    ```
+
+    If there are no errors, retrieve the pvc of the pod in `Pending` state.
+    ```bash
+    pvcName=$(oc get pod <osd_pod> -n <namespace> -o jsonpath='{.spec.volumes[*].persistentVolumeClaim.claimName}')
+    oc get pvc $pvcName -n <namespace> -o yaml
+    ```
+    Check for the health and events of the PVC for detailed information.
+
+2. PVC is in `Pending` state:
path: alerts/openshift-container-storage-operator/StorageAutoScalingFailed.md
user: login: travisn
body: Ok then I expect it will be rare
created_at: 2025-03-26T14:02:30Z
author_association: NONE
diff_hunk: @@ -0,0 +1,76 @@
+# StorageAutoScalingFailed
+
+## Meaning
+
+During automatic storage scaling, failures may occur due to various factors. In
+such instances, the `StorageAutoScalingFailed` alert is fired.
+
+## Impact
+
+The storage cannot be scaled at the moment.
+
+## Diagnosis
+
+There are multiple stages where the storage configurations might not have matched
+the requested storage parameters.
+
+1. OSD pod is in `Pending` state:
+    If it is in `Pending` state with the following logs, there is a high chance
+    it is due to the exhaustion of the resources.
+    ```bash
+    Warning  FailedScheduling  3m53s  default-scheduler  0/6 nodes are available: 1 Insufficient cpu, 2 node(s) didn't match Pod's node affinity/selector, 3 node(s) had untolerated taint {node-role.kubernetes.io/master: }. preemption: 0/6 nodes are available: 1 Insufficient cpu, 5 Preemption is not helpful for scheduling.
+    ```
+
+    Describe the pod in `Pending` state and check for the events.
+    ```bash
+    oc describe pod <osd_pod> -n <namespace>
+    ```
+
+    If there are no errors, retrieve the pvc of the pod in `Pending` state.
+    ```bash
+    pvcName=$(oc get pod <osd_pod> -n <namespace> -o jsonpath='{.spec.volumes[*].persistentVolumeClaim.claimName}')
+    oc get pvc $pvcName -n <namespace> -o yaml
+    ```
+    Check for the health and events of the PVC for detailed information.
+
+2. PVC is in `Pending` state:
+
+    ```bash
+    oc get pvc <pvc_name> -n <namespace> -o yaml
+    ```
+
+    Inspect the events of the pvc. Additionally, retrieve the pv of the pvc if
+    it is still in `pending` state.
+    ```bash
+    oc get pvc <pvc_name> -n <namespace> -o jsonpath='{.spec.volumeName}'
+    ```
+
+3. PV is in `Pending` state:
path: alerts/openshift-container-storage-operator/StorageAutoScalingFailed.md
user: login: travisn
body: While the PVC might be in pending state, I don't know if I've ever seen a PV in pending state. @parth-gr Have you ever seen that?
created_at: 2025-03-26T14:03:19Z
author_association: NONE
diff_hunk: @@ -0,0 +1,75 @@
+# StorageAutoScalingFailed
+
+## Meaning
+
+During automatic storage scaling, failures may occur due to various factors. In
+such instances, the `StorageAutoScalingFailed` alert is fired.
+
+## Impact
+
+The storage cannot be scaled at the moment.
+
+## Diagnosis
+
+There are multiple stages where the storage configurations might not have matched
+the requested storage parameters.
+
+1. OSD pod is in `Pending` state:
+    If it is in `Pending` state with the following logs, there is a high chance
+    it is due to the exhaustion of the resources.
+    ```bash
+    Warning  FailedScheduling  3m53s  default-scheduler  0/6 nodes are available: 1 Insufficient cpu, 2 node(s) didn't match Pod's node affinity/selector, 3 node(s) had untolerated taint {node-role.kubernetes.io/master: }. preemption: 0/6 nodes are available: 1 Insufficient cpu, 5 Preemption is not helpful for scheduling.
+    ```
+
+    Describe the pod in `Pending` state and check for the events.
+    ```bash
+    oc describe pod <osd_pod> -n <namespace>
+    ```
+
+    If there are no errors, retrieve the pvc of the pod in `Pending` state.
+    ```bash
+    pvcName=$(oc get pod <osd_pod> -n <namespace> -o jsonpath='{.spec.volumes[*].persistentVolumeClaim.claimName}')
+    oc get pvc $pvcName -n <namespace> -o yaml
+    ```
+    Check for the health and events of the PVC for detailed information.
+
+2. PVC is in `Pending` state:
+
+    ```bash
+    oc get pvc <pvc_name> -n <namespace> -o yaml
+    ```
+
+    Inspect the events of the pvc. Additionally, retrieve the pv of the pvc if
+    it is still in `pending` state.
+    ```bash
+    oc get pvc <pvc_name> -n <namespace> -o jsonpath='{.spec.volumeName}'
+    ```
+
+3. PV is in `Failed` state:
+    ```bash
+    oc get pv <pv_name> -n <namespace> -o yaml
+    ```
+    If the pv is in `Failed` state, there might be a storage issue. Check the
+    events of the pv for more information.
+
+## Mitigation
+
+1. If resources are depleted and pods start failing, allocate additional
+resources to the storage cluster. Possible solutions include adding more nodes
+or scaling up CPU and memory on existing nodes.
+- For instructions on adding nodes, please refer to this document: [Adding a node](https://docs.redhat.com/en/documentation/red_hat_openshift_data_foundation/4.17/html-single/scaling_storage/index#adding_a_node)
+
+2. Check upon the Ceph Storage: In case the OSDs are `down` or `out`, restart them
+and try again.
+
+3. During vertical scaling, if the pv takes time to resize, then the osds would not
path: alerts/openshift-container-storage-operator/StorageAutoScalingFailed.md
user: login: travisn
body: @parth-gr The only reason restarting the operator would help is if the Rook operator got stuck reconciling. So you're suggesting to restart the operator just in case it got stuck? That seems like a separate mitigation from restarting the OSDs, maybe it should be a separate bullet point?
created_at: 2025-03-26T14:07:18Z
author_association: NONE
diff_hunk: @@ -0,0 +1,75 @@
+# StorageAutoScalingFailed
+
+## Meaning
+
+During automatic storage scaling, failures may occur due to various factors. In
+such instances, the `StorageAutoScalingFailed` alert is fired.
+
+## Impact
+
+The storage cannot be scaled at the moment.
+
+## Diagnosis
+
+There are multiple stages where the storage configurations might not have matched
+the requested storage parameters.
+
+1. OSD pod is in `Pending` state:
+    If it is in `Pending` state with the following logs, there is a high chance
+    it is due to the exhaustion of the resources.
+    ```bash
+    Warning  FailedScheduling  3m53s  default-scheduler  0/6 nodes are available: 1 Insufficient cpu, 2 node(s) didn't match Pod's node affinity/selector, 3 node(s) had untolerated taint {node-role.kubernetes.io/master: }. preemption: 0/6 nodes are available: 1 Insufficient cpu, 5 Preemption is not helpful for scheduling.
+    ```
+
+    Describe the pod in `Pending` state and check for the events.
+    ```bash
+    oc describe pod <osd_pod> -n <namespace>
+    ```
+
+    If there are no errors, retrieve the pvc of the pod in `Pending` state.
+    ```bash
+    pvcName=$(oc get pod <osd_pod> -n <namespace> -o jsonpath='{.spec.volumes[*].persistentVolumeClaim.claimName}')
+    oc get pvc $pvcName -n <namespace> -o yaml
+    ```
+    Check for the health and events of the PVC for detailed information.
+
+2. PVC is in `Pending` state:
+
+    ```bash
+    oc get pvc <pvc_name> -n <namespace> -o yaml
+    ```
+
+    Inspect the events of the pvc. Additionally, retrieve the pv of the pvc if
+    it is still in `pending` state.
+    ```bash
+    oc get pvc <pvc_name> -n <namespace> -o jsonpath='{.spec.volumeName}'
+    ```
+
+3. PV is in `Failed` state:
+    ```bash
+    oc get pv <pv_name> -n <namespace> -o yaml
+    ```
+    If the pv is in `Failed` state, there might be a storage issue. Check the
+    events of the pv for more information.
+
+## Mitigation
+
+1. If resources are depleted and pods start failing, allocate additional
+resources to the storage cluster. Possible solutions include adding more nodes
+or scaling up CPU and memory on existing nodes.
+- For instructions on adding nodes, please refer to this document: [Adding a node](https://docs.redhat.com/en/documentation/red_hat_openshift_data_foundation/4.17/html-single/scaling_storage/index#adding_a_node)
+
+2. Check upon the Ceph Storage: In case the OSDs are `down` or `out`, restart them
+and try again.
+
+3. During vertical scaling, if the pv takes time to resize, then the osds would not
path: alerts/openshift-container-storage-operator/StorageAutoScalingFailed.md
user: login: parth-gr
body: > So you're suggesting to restart the operator just in case it got stuck?

Even if it's not stuck, Rook operator does have a watch on when the osd is resize or re-started.
The bug: https://issues.redhat.com/browse/DFBUGS-965
created_at: 2025-03-26T14:29:42Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,76 @@
+# StorageAutoScalingFailed
+
+## Meaning
+
+During automatic storage scaling, failures may occur due to various factors. In
+such instances, the `StorageAutoScalingFailed` alert is fired.
+
+## Impact
+
+The storage cannot be scaled at the moment.
+
+## Diagnosis
+
+There are multiple stages where the storage configurations might not have matched
+the requested storage parameters.
+
+1. OSD pod is in `Pending` state:
+    If it is in `Pending` state with the following logs, there is a high chance
+    it is due to the exhaustion of the resources.
+    ```bash
+    Warning  FailedScheduling  3m53s  default-scheduler  0/6 nodes are available: 1 Insufficient cpu, 2 node(s) didn't match Pod's node affinity/selector, 3 node(s) had untolerated taint {node-role.kubernetes.io/master: }. preemption: 0/6 nodes are available: 1 Insufficient cpu, 5 Preemption is not helpful for scheduling.
+    ```
+
+    Describe the pod in `Pending` state and check for the events.
+    ```bash
+    oc describe pod <osd_pod> -n <namespace>
+    ```
+
+    If there are no errors, retrieve the pvc of the pod in `Pending` state.
+    ```bash
+    pvcName=$(oc get pod <osd_pod> -n <namespace> -o jsonpath='{.spec.volumes[*].persistentVolumeClaim.claimName}')
+    oc get pvc $pvcName -n <namespace> -o yaml
+    ```
+    Check for the health and events of the PVC for detailed information.
+
+2. PVC is in `Pending` state:
+
+    ```bash
+    oc get pvc <pvc_name> -n <namespace> -o yaml
+    ```
+
+    Inspect the events of the pvc. Additionally, retrieve the pv of the pvc if
+    it is still in `pending` state.
+    ```bash
+    oc get pvc <pvc_name> -n <namespace> -o jsonpath='{.spec.volumeName}'
+    ```
+
+3. PV is in `Pending` state:
path: alerts/openshift-container-storage-operator/StorageAutoScalingFailed.md
user: login: parth-gr
body: Not both at the same time
created_at: 2025-03-26T14:30:16Z
author_association: MEMBER
diff_hunk: @@ -0,0 +1,81 @@
+# StorageAutoScalingFailed
+
+## Meaning
+
+During automatic storage scaling, failures may occur due to various factors. In
+such instances, the `StorageAutoScalingFailed` alert is fired.
+
+## Impact
+
+The storage cannot be scaled at the moment.
+
+## Diagnosis
+
+There are multiple stages where the storage configurations might not have matched
+the requested storage parameters.
+
+1. OSD pod is in `Pending` state:
+    If it is in `Pending` state with the following logs, there is a high chance
+    it is due to the exhaustion of the resources. Describe the pod in `Pending`
+    state and check for the events.
+    ```bash
+    oc describe pod <osd_pod> -n <namespace>
+    ```
+
+    If the output is similar to the example below, it indicates that resources
+    are exhausted:
+    ```bash
+    Warning  FailedScheduling  3m53s  default-scheduler  0/6 nodes are available: 1 Insufficient cpu, 2 node(s) didn't match Pod's node affinity/selector, 3 node(s) had untolerated taint {node-role.kubernetes.io/master: }. preemption: 0/6 nodes are available: 1 Insufficient cpu, 5 Preemption is not helpful for scheduling.
+    ```
+
+    If there are no errors, retrieve the pvc of the pod in `Pending` state.
+    ```bash
+    pvcName=$(oc get pod <osd_pod> -n <namespace> -o jsonpath='{.spec.volumes[*].persistentVolumeClaim.claimName}')
+    oc get pvc $pvcName -n <namespace> -o yaml
+    ```
+    Check for the health and events of the PVC for detailed information.
+
+2. PVC is in `Pending` state:
+
+    ```bash
+    oc get pvc <pvc_name> -n <namespace> -o yaml
+    ```
+
+    Inspect the events of the pvc. Additionally, retrieve the pv of the pvc if
+    it is still in `pending` state.
+    ```bash
+    oc get pvc <pvc_name> -n <namespace> -o jsonpath='{.spec.volumeName}'
+    ```
+
+3. PV is in `Pending` state:
+    ```bash
+    oc get pv <pv_name> -n <namespace> -o yaml
+    ```
+    If the pv is in `Pending` state, there might be a storage issue. Check the
+    events of the pv for more information.
+
+## Mitigation
+
+1. If resources are depleted and pods start failing, allocate additional
+resources to the storage cluster. Possible solutions include adding more nodes
+or scaling up CPU and memory on existing nodes.
+- For instructions on adding nodes, please refer to this document: https://docs.redhat.com/en/documentation/red_hat_openshift_data_foundation/<odf-version>/html-single/scaling_storage/index#adding_a_node
path: alerts/openshift-container-storage-operator/StorageAutoScalingFailed.md
user: login: travisn
body: Will this raw text be shown to the user or will it be rendered in some way? For example, if I change to the markdown rendering, the `<odf-version>` disappears and the user can't see that they need to insert the version to the URL. 

But is there any way to have a URL that works by default so they don't have to replace a variable every time? 
created_at: 2025-03-26T17:24:50Z
author_association: NONE
diff_hunk: @@ -0,0 +1,81 @@
+# StorageAutoScalingFailed
+
+## Meaning
+
+During automatic storage scaling, failures may occur due to various factors. In
+such instances, the `StorageAutoScalingFailed` alert is fired.
+
+## Impact
+
+The storage cannot be scaled at the moment.
+
+## Diagnosis
+
+There are multiple stages where the storage configurations might not have matched
+the requested storage parameters.
+
+1. OSD pod is in `Pending` state:
+    If it is in `Pending` state with the following logs, there is a high chance
+    it is due to the exhaustion of the resources. Describe the pod in `Pending`
+    state and check for the events.
+    ```bash
+    oc describe pod <osd_pod> -n <namespace>
+    ```
+
+    If the output is similar to the example below, it indicates that resources
+    are exhausted:
+    ```bash
+    Warning  FailedScheduling  3m53s  default-scheduler  0/6 nodes are available: 1 Insufficient cpu, 2 node(s) didn't match Pod's node affinity/selector, 3 node(s) had untolerated taint {node-role.kubernetes.io/master: }. preemption: 0/6 nodes are available: 1 Insufficient cpu, 5 Preemption is not helpful for scheduling.
+    ```
+
+    If there are no errors, retrieve the pvc of the pod in `Pending` state.
+    ```bash
+    pvcName=$(oc get pod <osd_pod> -n <namespace> -o jsonpath='{.spec.volumes[*].persistentVolumeClaim.claimName}')
+    oc get pvc $pvcName -n <namespace> -o yaml
+    ```
+    Check for the health and events of the PVC for detailed information.
+
+2. PVC is in `Pending` state:
+
+    ```bash
+    oc get pvc <pvc_name> -n <namespace> -o yaml
+    ```
+
+    Inspect the events of the pvc. Additionally, retrieve the pv of the pvc if
+    it is still in `pending` state.
+    ```bash
+    oc get pvc <pvc_name> -n <namespace> -o jsonpath='{.spec.volumeName}'
+    ```
+
+3. PV is in `Pending` state:
+    ```bash
+    oc get pv <pv_name> -n <namespace> -o yaml
+    ```
+    If the pv is in `Pending` state, there might be a storage issue. Check the
+    events of the pv for more information.
+
+## Mitigation
+
+1. If resources are depleted and pods start failing, allocate additional
+resources to the storage cluster. Possible solutions include adding more nodes
+or scaling up CPU and memory on existing nodes.
+- For instructions on adding nodes, please refer to this document: https://docs.redhat.com/en/documentation/red_hat_openshift_data_foundation/<odf-version>/html-single/scaling_storage/index#adding_a_node
path: alerts/openshift-container-storage-operator/StorageAutoScalingFailed.md
user: login: ShravaniVangur
body: Just verified that the `latest` tag always points to the latest release notes. Added that to the link instead of 
`<odf-version> `.
created_at: 2025-03-26T17:54:10Z
author_association: CONTRIBUTOR

PR comments:
user: login: openshift-ci[bot]
created_at: 2025-03-18T08:03:38Z
body: Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with `/test all`
author_association: CONTRIBUTOR
user: login: parth-gr
created_at: 2025-03-25T13:58:21Z
body: @travisn @Madhu-1 PTAL^^
author_association: MEMBER
user: login: openshift-ci[bot]
created_at: 2025-03-26T19:06:15Z
body: @ShravaniVangur: all tests **passed!**


[Full PR test history](https://prow.ci.openshift.org/pr-history?org=openshift&repo=runbooks&pr=243). [Your PR dashboard](https://prow.ci.openshift.org/pr?query=is:pr+state:open+author:ShravaniVangur).

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
author_association: CONTRIBUTOR
user: login: parth-gr
created_at: 2025-03-27T06:35:22Z
body: cc @agarwal-mudit can we get this merge
author_association: MEMBER
user: login: parth-gr
created_at: 2025-03-27T08:13:55Z
body: /lgtm
author_association: MEMBER
user: login: parth-gr
created_at: 2025-03-27T08:15:49Z
body: /approve
author_association: MEMBER
user: login: openshift-ci[bot]
created_at: 2025-03-27T09:59:03Z
body: [APPROVALNOTIFIER] This PR is **APPROVED**

This pull-request has been approved by: *<a href="https://github.com/openshift/runbooks/pull/243#pullrequestreview-2720629216" title="Approved">agarwal-mudit</a>*, *<a href="https://github.com/openshift/runbooks/pull/243#issuecomment-2757108818" title="Approved">parth-gr</a>*, *<a href="https://github.com/openshift/runbooks/pull/243#" title="Author self-approved">ShravaniVangur</a>*, *<a href="https://github.com/openshift/runbooks/pull/243#pullrequestreview-2718191255" title="Approved">travisn</a>*

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=openshift%2Frunbooks).

The pull request process is described [here](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process)

<details >
Needs approval from an approver in each of these files:

- ~~[alerts/openshift-container-storage-operator/OWNERS](https://github.com/openshift/runbooks/blob/master/alerts/openshift-container-storage-operator/OWNERS)~~ [agarwal-mudit]

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":[]} -->
author_association: CONTRIBUTOR