Repository: FalkorDB/falkordb-dbaas
Pull request #157: oom alerts
Author: login: dudizimber
Created: 2025-03-05T08:37:12Z | Merged: 2025-03-06T11:41:38Z

fix #138 

<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit

- **New Features**
	- Introduced an automated process to validate alert configurations when relevant updates occur.
	- Added enhanced monitoring for container terminations due to out-of-memory events.
	- Implemented an automated validation script to ensure reliable alerting outcomes.
	- New GitHub Actions workflow for testing alert rules on pull requests.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Linked issue #138: Pods OOMs are not triggering alerts

Diff:
diff --git a/.github/workflows/alert-rules-test.yaml b/.github/workflows/alert-rules-test.yaml
new file mode 100644
index 000000000..470f0529f
--- /dev/null
+++ b/.github/workflows/alert-rules-test.yaml
@@ -0,0 +1,33 @@
+name: Test alert rules
+on:
+  workflow_dispatch:
+  pull_request:
+    branches:
+      - dev
+      - main
+    paths:
+      - 'observability/rules/**'
+      - 'observability/tests/**'
+
+concurrency:
+  group: alert-rules-${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+      
+      - name: Install PromTool
+        run: |
+          VERSION=$(curl -Ls https://api.github.com/repos/prometheus/prometheus/releases/latest | jq ".tag_name" | xargs | cut -c2-)
+          wget -qO- "https://github.com/prometheus/prometheus/releases/download/v${VERSION}/prometheus-$VERSION.linux-amd64.tar.gz" \
+            | tar xvzf - "prometheus-$VERSION.linux-amd64"/promtool --strip-components=1
+          sudo mv promtool /usr/local/bin/promtool
+          promtool --version
+
+      - name: Run tests
+        run: ./scripts/run_alerts_tests.sh
+
diff --git a/observability/rules/containeroom.rules.yml b/observability/rules/containeroom.rules.yml
new file mode 100644
index 000000000..6aedbc6d1
--- /dev/null
+++ b/observability/rules/containeroom.rules.yml
@@ -0,0 +1,20 @@
+apiVersion: operator.victoriametrics.com/v1beta1
+kind: VMRule
+metadata:
+  namespace: observability
+  name: k8s.rules.containeroom
+spec:
+  groups:
+    - name: k8s.rules.container_oom
+      rules:
+        - alert: ContainerOOMKilled
+          expr: |
+            group by (pod, cluster, namespace) (changes(kube_pod_container_status_restarts_total[5m]) > 0)
+            and
+            group by (pod, cluster, namespace) (kube_pod_container_status_last_terminated_reason{reason="OOMKilled"} == 1)
+          for: 0m
+          labels:
+            severity: critical
+          annotations:
+            summary: "Container OOMKilled"
+            description: "Pod {{ $labels.pod }} in namespace {{ $labels.namespace }} was terminated due to OOMKilled."
diff --git a/scripts/run_alerts_tests.sh b/scripts/run_alerts_tests.sh
new file mode 100755
index 000000000..06f1346ec
--- /dev/null
+++ b/scripts/run_alerts_tests.sh
@@ -0,0 +1,41 @@
+#!/bin/bash
+
+# Check that promtool is installed
+if ! command -v promtool &> /dev/null
+then
+    echo "promtool could not be found"
+    exit 1
+fi
+
+# skip if there are no tests
+if [ ! -d observability/rules/tests ] || [ -z "$( ls -A observability/rules/tests )" ]; then
+  echo "No tests found"
+  exit 0
+fi
+
+mkdir -p observability/rules/tests/rules
+
+# For each file under observability/rules directory, create a temp yaml file and run the tests under observability/rules/tests directory
+for file in observability/rules/*.yml
+do
+  # Create temp rule file under observability/rules/tests directory
+  cp $file observability/rules/tests/rules/
+  yq eval -i '.groups = .spec.groups | del(.apiVersion, .kind, .metadata, .groups[].params, .spec)' observability/rules/tests/rules/$(basename $file)
+done
+
+errors=0
+for test_file in observability/rules/tests/*.test.yml
+do
+  promtool test rules --debug $test_file
+  if [ $? -ne 0 ]; then
+    errors=$((errors+1))
+  fi
+done
+
+# Clean up temp files
+rm -rf observability/rules/tests/rules
+
+if [ $errors -ne 0 ]; then
+  echo "Tests failed"
+  exit 1
+fi
\ No newline at end of file


Reviews:
user: login: gemini-code-assist[bot]
body: Hello @dudizimber, I'm Gemini Code Assist[^1]! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

## Summary of Changes
This pull request addresses issue #138 by adding an alert for when a container is OOMKilled in Kubernetes. I've added a new alert rule in `observability/rules/containeroom.rules.yml` that triggers when `kube_pod_container_status_last_terminated_reason` metric has `reason="OOMKilled"`. I've also included a test for this rule in `observability/rules/tests/containeroom.rules.test.yml`. Finally, I've added a new workflow file `.github/workflows/alert-rules-test.yaml` to test the alert rules and modified the `scripts/run_alerts_tests.sh` script to accommodate the new rule and test.

### Highlights
* **New Alert Rule**: Added a new alert rule `ContainerOOMKilled` to detect OOMKilled containers.
* **Alert Rule Test**: Created a test case for the new alert rule to ensure it functions correctly.
* **Workflow for Alert Rule Testing**: Implemented a GitHub Actions workflow to automatically test alert rules on pull requests.
* **Testing Script Modification**: Modified the alert rules testing script to handle the new alert rule and its test file, including dynamically generating temporary rule files for testing.

### Changelog
<details>
<summary>Click here to see the changelog</summary>

* **.github/workflows/alert-rules-test.yaml**
  * Added a new workflow to test alert rules on pull requests and workflow dispatch events.
  * The workflow checks out the repository, installs Promtool, and runs the alert tests.
* **observability/rules/containeroom.rules.yml**
  * Added a new alert rule `ContainerOOMKilled` to detect when a container is terminated due to OOMKilled.
  * The alert triggers when the `kube_pod_container_status_last_terminated_reason` metric indicates an OOMKilled reason.
* **observability/rules/tests/containeroom.rules.test.yml**
  * Created a test file for the `ContainerOOMKilled` alert rule.
  * The test defines an input series that simulates a container being OOMKilled and verifies that the alert is triggered with the expected labels and annotations.
* **scripts/run_alerts_tests.sh**
  * Modified the script to dynamically create temporary rule files for testing.
  * The script now copies the rule files, extracts the groups from the spec, and then runs the promtool test rules command.
  * Added error handling to exit with an error code if any tests fail.

</details>


<details>

<summary><b>Using Gemini Code Assist</b></summary>
<br>

The full guide for Gemini Code Assist can be found on our [documentation page](https://developers.google.com/gemini-code-assist/docs/review-github-code), here are some quick tips.

<b>Invoking Gemini</b>

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either `/gemini <command>` or `@gemini-code-assist <command>`. Below is a summary of the supported commands.

Feature | Command | Description
--- | --- | ---
Code Review | `/gemini review` | Performs a code review for the current pull request in its current state.
Pull Request Summary | `/gemini summary` | Provides a summary of the current pull request in its current state.
Comment | @gemini-code-assist | Responds in comments when explicitly tagged, both in issue comments and review comments.
Help | `/gemini help` | Displays a list of available commands.

<b>Customization</b>

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a `.gemini/` folder in the base of the repository. Detailed instructions can be found [here](https://developers.google.com/gemini-code-assist/docs/review-github-code).

<b>Limitations & Feedback</b>

Gemini Code Assist is currently in preview and may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with :thumbsup: and :thumbsdown: on @gemini-code-assist comments to provide feedback.

</details>

<b>You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the [Gemini Code Assist IDE Extension](https://cloud.google.com/products/gemini/code-assist).</b>


[^1]: Review the [Privacy Notices](https://policies.google.com/privacy), [Generative AI Prohibited Use Policy](https://policies.google.com/terms/generative-ai/use-policy), [Terms of Service](https://policies.google.com/terms), and learn how to configure Gemini Code Assist in GitHub [here](https://developers.google.com/gemini-code-assist/docs/review-github-code). Gemini can make mistakes, so double check it and [use code with caution](https://support.google.com/legal/answer/13505487).

---

> A container's plight, OOM's dark kiss,
> Memory exhausted, a system's miss.
> Alerts arise, a watchful eye,
> To catch the moment, before containers die.


state: COMMENTED
author_association: NONE
submitted_at: 2025-03-05T08:37:23Z
user: login: gemini-code-assist[bot]
body: ## Code Review
This pull request introduces a new alert rule for detecting container OOMKilled events and adds corresponding tests. The changes also include a workflow for testing alert rules and a script to run the tests. Overall, the changes look good, but there are a few areas that could be improved.

### Summary of Findings
* **Missing Namespace in Test Annotation**: The test annotation description in `observability/rules/tests/containeroom.rules.test.yml` does not include the namespace variable, which could lead to confusion when the alert is triggered in different namespaces. While the namespace is present in the labels, it's best practice to include it in the description as well.
* **Incomplete yq command**: The `yq` command in `scripts/run_alerts_tests.sh` removes the `spec` field entirely, which is not the intention. The intention is to remove the `spec` field's contents, but not the `spec` field itself.

### Merge Readiness
The pull request is almost ready for merging. The identified issues should be addressed before merging to ensure the alert rule and tests function as expected. I am unable to approve this pull request, and recommend that others review and approve this code before merging.
state: CHANGES_REQUESTED
author_association: NONE
submitted_at: 2025-03-05T08:37:34Z
user: login: coderabbitai[bot]
body: **Actionable comments posted: 1**

<details>
<summary>🧹 Nitpick comments (3)</summary><blockquote>

<details>
<summary>observability/rules/containeroom.rules.yml (1)</summary><blockquote>

`9-12`: **Alert Rule Definition Looks Sound.**  
The alert `ContainerOOMKilled` is defined with the expression checking for OOM termination (`== 1`) and a hold time of `0m`. Consider whether a non-zero duration might help filter transient conditions, based on your environment’s behavior.

</blockquote></details>
<details>
<summary>.github/workflows/alert-rules-test.yaml (2)</summary><blockquote>

`20-22`: **Repository Checkout Step and Formatting.**  
The repository checkout step is correctly implemented using `actions/checkout@v4`. Note that line 22 appears to have trailing spaces; cleaning these up would improve file formatting.

<details>
<summary>🧰 Tools</summary>

<details>
<summary>🪛 YAMLlint (1.35.1)</summary>

[error] 22-22: trailing spaces

(trailing-spaces)

</details>

</details>

---

`31-33`: **Run Tests Step is Clear.**  
The step to execute the alert tests via `./scripts/run_alerts_tests.sh` is straightforward. Removing extra blank lines (as noted by YAML lint) would help maintain a cleaner file format.

<details>
<summary>🧰 Tools</summary>

<details>
<summary>🪛 YAMLlint (1.35.1)</summary>

[warning] 33-33: too many blank lines

(1 > 0) (empty-lines)

</details>

</details>

</blockquote></details>

</blockquote></details>

<details>
<summary>📜 Review details</summary>

**Configuration used: CodeRabbit UI**
**Review profile: CHILL**
**Plan: Pro**

<details>
<summary>📥 Commits</summary>

Reviewing files that changed from the base of the PR and between 65ebab2302ce456f08cff9972a4fc662c2ebda1c and ce41c8c0bc041c04e3e279ff1056f423b3ec682b.

</details>

<details>
<summary>📒 Files selected for processing (4)</summary>

* `.github/workflows/alert-rules-test.yaml` (1 hunks)
* `observability/rules/containeroom.rules.yml` (1 hunks)
* `observability/rules/tests/containeroom.rules.test.yml` (1 hunks)
* `scripts/run_alerts_tests.sh` (1 hunks)

</details>

<details>
<summary>🧰 Additional context used</summary>

<details>
<summary>🪛 YAMLlint (1.35.1)</summary>

<details>
<summary>.github/workflows/alert-rules-test.yaml</summary>

[error] 22-22: trailing spaces

(trailing-spaces)

---

[warning] 33-33: too many blank lines

(1 > 0) (empty-lines)

</details>

</details>
<details>
<summary>🪛 Shellcheck (0.10.0)</summary>

<details>
<summary>scripts/run_alerts_tests.sh</summary>

[warning] 17-17: Quote this to prevent word splitting.

(SC2046)

</details>

</details>

</details>

<details>
<summary>⏰ Context from checks skipped due to timeout of 90000ms (1)</summary>

* GitHub Check: Analyze (javascript-typescript)

</details>

<details>
<summary>🔇 Additional comments (15)</summary><blockquote>

<details>
<summary>observability/rules/tests/containeroom.rules.test.yml (4)</summary>

`1-3`: **Rule File Reference is Correct.**  
The `rule_files` list correctly references the local test rule file (`./rules/containeroom.rules.yml`).

---

`4-5`: **Evaluation Interval is Set.**  
The `evaluation_interval` of 10 minutes is clearly defined. Please verify that this interval meets your alert evaluation requirements.

---

`6-11`: **Test Input Series and Interval Structured Well.**  
The test configuration defines a 1-minute interval and provides an `input_series` simulating an OOM event. Ensure that the `values` string ("1 1 1 1 1") correctly mimics the time-series data expected by your alerting system.

---

`12-24`: **Alert Rule Test Specification is Comprehensive.**  
The `alert_rule_test` section specifies the evaluation time, expected alert name, and detailed expected labels and annotations. Please confirm that these expected values match the labels and annotations produced by the actual alert rule at runtime.

</details>
<details>
<summary>observability/rules/containeroom.rules.yml (4)</summary>

`1-2`: **API Version and Kind Declaration are Clear.**  
The VMRule is correctly declared with `apiVersion: operator.victoriametrics.com/v1beta1` and `kind: VMRule`.

---

`3-5`: **Metadata is Defined Appropriately.**  
The metadata block accurately sets the namespace to `observability` and defines a consistent rule name.

---

`6-8`: **Rule Group Configuration is Valid.**  
The rule group `k8s.rules.container_oom` is properly structured within the spec.

---

`13-17`: **Labels and Annotations are Correctly Set.**  
The rule assigns a critical severity and dynamically constructs the alert description using label values. This setup should effectively convey alert context.

</details>
<details>
<summary>scripts/run_alerts_tests.sh (4)</summary>

`1-8`: **PromTool Installation Check is Good.**  
The script properly verifies the presence of `promtool` and exits with an error message if it is missing, which is essential before proceeding with tests.

---

`10-10`: **Directory Creation is Appropriate.**  
Using `mkdir -p` ensures that the temporary directory for test rules is created without issues even if it already exists.

---

`20-27`: **Alert Rule Testing Loop Looks Good.**  
The script iterates over each `.test.yml` file and uses `promtool test rules` to run tests, while counting any errors. This pattern is effective for summarizing test outcomes.

---

`29-35`: **Cleanup and Error Reporting are Effective.**  
After test execution, the temporary files are removed and the script exits with a non-zero status if any errors occurred, correctly signaling test failures.

</details>
<details>
<summary>.github/workflows/alert-rules-test.yaml (3)</summary>

`1-11`: **Workflow Trigger Configuration is Verified.**  
The workflow is set to trigger on manual dispatch (`workflow_dispatch`) and pull requests affecting the specified paths on the `dev` and `main` branches. Ensure that these path filters encompass all necessary alert rule and test file changes.

---

`12-15`: **Concurrency Settings are Effective.**  
Using dynamic variables for the concurrency group and enabling cancellation of in-progress runs are good practices for efficient CI execution.

---

`23-29`: **PromTool Installation Step is Comprehensive.**  
The installation step fetches the latest release, extracts `promtool`, and verifies its version. Consider adding error handling for potential failures during the download or extraction processes.

</details>

</blockquote></details>

</details>

<!-- This is an auto-generated comment by CodeRabbit for review status -->
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-05T08:39:58Z
user: login: coderabbitai[bot]
body: **Actionable comments posted: 1**

<details>
<summary>🧹 Nitpick comments (1)</summary><blockquote>

<details>
<summary>scripts/run_alerts_tests.sh (1)</summary><blockquote>

`27-29`: **Refactor Suggestion: Quote `$test_file` in Promtool Invocation**

On line 29, the `$test_file` variable is passed to the `promtool` command without quotes, which may cause issues if the filename contains spaces.  

**Suggested Fix:** Enclose `$test_file` in quotes as shown below:  

```diff
-  promtool test rules --debug $test_file
+  promtool test rules --debug "$test_file"
```

</blockquote></details>

</blockquote></details>

<details>
<summary>📜 Review details</summary>

**Configuration used: CodeRabbit UI**
**Review profile: CHILL**
**Plan: Pro**

<details>
<summary>📥 Commits</summary>

Reviewing files that changed from the base of the PR and between ce41c8c0bc041c04e3e279ff1056f423b3ec682b and f1b99aa6a58d82a5476905860dcd5af7ccf455ec.

</details>

<details>
<summary>📒 Files selected for processing (2)</summary>

* `observability/rules/containeroom.rules.yml` (1 hunks)
* `scripts/run_alerts_tests.sh` (1 hunks)

</details>

<details>
<summary>🚧 Files skipped from review as they are similar to previous changes (1)</summary>

* observability/rules/containeroom.rules.yml

</details>

<details>
<summary>🧰 Additional context used</summary>

<details>
<summary>🪛 Shellcheck (0.10.0)</summary>

<details>
<summary>scripts/run_alerts_tests.sh</summary>

[warning] 23-23: Quote this to prevent word splitting.

(SC2046)

</details>

</details>

</details>

<details>
<summary>🔇 Additional comments (1)</summary><blockquote>

<details>
<summary>scripts/run_alerts_tests.sh (1)</summary>

`21-23`: **Refactor Suggestion: Quote Variable Expansions in File Loop**

In the file loop (lines 21–23), the `$file` variable and its basename are used without quotes. This could lead to word-splitting issues if filenames contain spaces. It is best practice to quote such expansions to ensure reliable handling of filenames.  

**Suggested Fix:** Update the `cp` and `yq` commands as follows:  

```diff
-  cp $file observability/rules/tests/rules/
-  yq eval -i '.groups = .spec.groups | del(.apiVersion, .kind, .metadata, .groups[].params, .spec)' observability/rules/tests/rules/$(basename $file)
+  cp "$file" observability/rules/tests/rules/
+  yq eval -i '.groups = .spec.groups | del(.apiVersion, .kind, .metadata, .groups[].params, .spec)' observability/rules/tests/rules/"$(basename "$file")"
```  

This change also addresses the static analysis warning (SC2046) regarding unquoted command substitutions.

<details>
<summary>🧰 Tools</summary>

<details>
<summary>🪛 Shellcheck (0.10.0)</summary>

[warning] 23-23: Quote this to prevent word splitting.

(SC2046)

</details>

</details>

</details>

</blockquote></details>

</details>

<!-- This is an auto-generated comment by CodeRabbit for review status -->
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-06T09:26:57Z
user: login: coderabbitai[bot]
body: **Actionable comments posted: 0**

<details>
<summary>🧹 Nitpick comments (2)</summary><blockquote>

<details>
<summary>scripts/run_alerts_tests.sh (2)</summary><blockquote>

`19-24`: **File Loop Variable Quotation Improvement:**  
To prevent potential issues with filenames that contain spaces or special characters, it is advisable to quote the variable expansions in the loop. Please update the `cp` and `yq` commands accordingly.  
Suggested diff:  
```diff
-  cp $file observability/rules/tests/rules/
-  yq eval -i '.groups = .spec.groups | del(.apiVersion, .kind, .metadata, .groups[].params, .spec)' observability/rules/tests/rules/$(basename $file)
+  cp "$file" observability/rules/tests/rules/
+  yq eval -i '.groups = .spec.groups | del(.apiVersion, .kind, .metadata, .groups[].params, .spec)' observability/rules/tests/rules/"$(basename "$file")"
```

<details>
<summary>🧰 Tools</summary>

<details>
<summary>🪛 Shellcheck (0.10.0)</summary>

[warning] 23-23: Quote this to prevent word splitting.

(SC2046)

</details>

</details>

---

`26-33`: **Test File Loop Quotation Improvement:**  
For robustness and to avoid unintended word splitting, consider quoting the `$test_file` variable when passing it to the `promtool` command.  
Suggested diff:  
```diff
-  promtool test rules --debug $test_file
+  promtool test rules --debug "$test_file"
```

</blockquote></details>

</blockquote></details>

<details>
<summary>📜 Review details</summary>

**Configuration used: CodeRabbit UI**
**Review profile: CHILL**
**Plan: Pro**

<details>
<summary>📥 Commits</summary>

Reviewing files that changed from the base of the PR and between f1b99aa6a58d82a5476905860dcd5af7ccf455ec and 0160d4f8548b8c9e865960e3433114d6205b4cc0.

</details>

<details>
<summary>📒 Files selected for processing (1)</summary>

* `scripts/run_alerts_tests.sh` (1 hunks)

</details>

<details>
<summary>🧰 Additional context used</summary>

<details>
<summary>🪛 Shellcheck (0.10.0)</summary>

<details>
<summary>scripts/run_alerts_tests.sh</summary>

[warning] 23-23: Quote this to prevent word splitting.

(SC2046)

</details>

</details>

</details>

<details>
<summary>🔇 Additional comments (3)</summary><blockquote>

<details>
<summary>scripts/run_alerts_tests.sh (3)</summary>

`1-8`: **Promtool Check Section:**  
The script correctly checks for the presence of `promtool` using `command -v` and exits with a helpful error message if it's not installed.

---

`10-14`: **Test Directory Existence Check:**  
The condition now correctly uses the `-z` operator to detect if the `observability/rules/tests` directory is empty (or doesn’t exist). This aligns with the intended behavior to only skip tests when no test files are present.

---

`35-41`: **Cleanup and Error Handling:**  
The cleanup process (removing the temporary rules directory) and the subsequent error handling mechanism are implemented correctly.

</details>

</blockquote></details>

</details>

<!-- This is an auto-generated comment by CodeRabbit for review status -->
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-06T11:16:30Z
user: login: AviAvni
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-06T11:32:02Z

Review comments:
diff_hunk: @@ -0,0 +1,35 @@
+#!/bin/bash
+
+# Check that promtool is installed
+if ! command -v promtool &> /dev/null
+then
+    echo "promtool could not be found"
+    exit 1
+fi
+
+mkdir -p observability/rules/tests/rules
+
+# For each file under observability/rules directory, create a temp yaml file and run the tests under observability/rules/tests directory
+for file in observability/rules/*.yml
+do
+  # Create temp rule file under observability/rules/tests directory
+  cp $file observability/rules/tests/rules/
+  yq eval -i '.groups = .spec.groups | del(.apiVersion, .kind, .metadata, .groups[].params, .spec)' observability/rules/tests/rules/$(basename $file)
path: scripts/run_alerts_tests.sh
user: login: gemini-code-assist[bot]
body: ![high](https://img.shields.io/badge/priority-high-orange)

The `yq` command removes the `spec` field entirely.  Instead, remove the contents of the `spec` field, but not the `spec` field itself.  This will prevent errors when running the tests.

```suggestion
  yq eval -i '.groups = .spec.groups | del(.apiVersion, .kind, .metadata, .groups[].params) | .spec = {}' observability/rules/tests/rules/$(basename $file)
```
created_at: 2025-03-05T08:37:34Z
author_association: NONE
diff_hunk: @@ -0,0 +1,24 @@
+rule_files:
+  - "./rules/containeroom.rules.yml"
+
+evaluation_interval: 10m
+
+tests:
+  - interval: 1m
+    input_series:
+      - series: 'kube_pod_container_status_last_terminated_reason{reason="OOMKilled", pod="test-pod", namespace="test-ns", container="test-container"}'
+        values: "1 1 1 1 1"
+
+    alert_rule_test:
+      - eval_time: 1m
+        alertname: ContainerOOMKilled
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              pod: "test-pod"
+              namespace: "test-ns"
+              container: "test-container"
+              reason: "OOMKilled"
+            exp_annotations:
+              summary: "Container OOMKilled"
+              description: "Container test-pod (test-container) in namespace test-ns was terminated due to OOMKilled."
path: observability/rules/tests/containeroom.rules.test.yml
user: login: gemini-code-assist[bot]
body: ![medium](https://img.shields.io/badge/priority-medium-yellow)

Consider including the namespace variable in the description for better clarity. While the namespace is present in the labels, it's best practice to include it in the description as well to avoid confusion when the alert is triggered in different namespaces.

```
              description: "Container test-pod (test-container) in namespace test-ns was terminated due to OOMKilled in namespace test-ns."
```
created_at: 2025-03-05T08:37:34Z
author_association: NONE
diff_hunk: @@ -0,0 +1,35 @@
+#!/bin/bash
+
+# Check that promtool is installed
+if ! command -v promtool &> /dev/null
+then
+    echo "promtool could not be found"
+    exit 1
+fi
+
+mkdir -p observability/rules/tests/rules
+
+# For each file under observability/rules directory, create a temp yaml file and run the tests under observability/rules/tests directory
+for file in observability/rules/*.yml
+do
+  # Create temp rule file under observability/rules/tests directory
+  cp $file observability/rules/tests/rules/
+  yq eval -i '.groups = .spec.groups | del(.apiVersion, .kind, .metadata, .groups[].params, .spec)' observability/rules/tests/rules/$(basename $file)
+done
path: scripts/run_alerts_tests.sh
user: login: coderabbitai[bot]
body: _⚠️ Potential issue_

**File Loop Processing Improvement.**  
The loop correctly copies each YAML file from `observability/rules` into the test directory and modifies it with `yq` for testing purposes. However, to prevent issues with filenames containing spaces, wrap the command substitution in quotes. For example:  
```diff
-  yq eval -i '.groups = .spec.groups | del(.apiVersion, .kind, .metadata, .groups[].params, .spec)' observability/rules/tests/rules/$(basename $file)
+  yq eval -i '.groups = .spec.groups | del(.apiVersion, .kind, .metadata, .groups[].params, .spec)' observability/rules/tests/rules/"$(basename "$file")"
```  



<!-- suggestion_start -->

<details>
<summary>📝 Committable suggestion</summary>

> ‼️ **IMPORTANT**
> Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

`````suggestion
for file in observability/rules/*.yml
do
  # Create temp rule file under observability/rules/tests directory
  cp $file observability/rules/tests/rules/
  yq eval -i '.groups = .spec.groups | del(.apiVersion, .kind, .metadata, .groups[].params, .spec)' observability/rules/tests/rules/"$(basename "$file")"
done
`````

</details>

<!-- suggestion_end -->

<details>
<summary>🧰 Tools</summary>

<details>
<summary>🪛 Shellcheck (0.10.0)</summary>

[warning] 17-17: Quote this to prevent word splitting.

(SC2046)

</details>

</details>

<!-- This is an auto-generated comment by CodeRabbit -->
created_at: 2025-03-05T08:39:57Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,41 @@
+#!/bin/bash
+
+# Check that promtool is installed
+if ! command -v promtool &> /dev/null
+then
+    echo "promtool could not be found"
+    exit 1
+fi
+
+# skip if there are no tests
+if [ ! -d observability/rules/tests ] || [ -n "$( ls -A observability/rules/tests )" ]; then
+  echo "No tests found"
+  exit 0
+fi
path: scripts/run_alerts_tests.sh
user: login: coderabbitai[bot]
body: _💡 Verification agent_

<details>
<summary>❓ Verification inconclusive</summary>

**Critical Issue: Incorrect Test-Skip Condition**

The condition on line 11 is intended to skip the tests if the tests directory does not exist or is empty. However, it incorrectly uses the `-n` operator, which checks for a non-empty string. This means that when the directory is actually non-empty, the tests will be skipped unexpectedly.  
  
**Suggested Fix:** Replace `-n` with `-z` to correctly detect an empty directory.  
  
```diff
-if [ ! -d observability/rules/tests ] || [ -n "$( ls -A observability/rules/tests )" ]; then
+if [ ! -d observability/rules/tests ] || [ -z "$( ls -A observability/rules/tests )" ]; then
```

---



</details>

**Fix Required: Incorrect Test-Skip Condition in `scripts/run_alerts_tests.sh`**

It has been confirmed that the current code mistakenly uses the `-n` operator to check if the tests directory is non-empty. This inadvertently causes the tests to be skipped when the directory contains files—contrary to the intended behavior of skipping tests only when the directory is absent or empty.

- **File:** `scripts/run_alerts_tests.sh` (Lines 10-14)
- **Issue:** The condition uses `-n "$( ls -A observability/rules/tests )"` which returns true for non-empty directories.
- **Suggested Fix:** Replace `-n` with `-z` so the condition correctly detects an empty directory.

**Diff:**
```diff
-if [ ! -d observability/rules/tests ] || [ -n "$( ls -A observability/rules/tests )" ]; then
+if [ ! -d observability/rules/tests ] || [ -z "$( ls -A observability/rules/tests )" ]; then
```



<!-- suggestion_start -->

<details>
<summary>📝 Committable suggestion</summary>

> ‼️ **IMPORTANT**
> Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

`````suggestion
# skip if there are no tests
if [ ! -d observability/rules/tests ] || [ -z "$( ls -A observability/rules/tests )" ]; then
  echo "No tests found"
  exit 0
fi
`````

</details>

<!-- suggestion_end -->

<!-- This is an auto-generated comment by CodeRabbit -->
created_at: 2025-03-06T09:26:56Z
author_association: CONTRIBUTOR

PR comments:
user: login: coderabbitai[bot]
created_at: 2025-03-05T08:37:21Z
body: <!-- This is an auto-generated comment: summarize by coderabbit.ai -->
<!-- walkthrough_start -->

## Walkthrough
This pull request introduces three new files. A GitHub Actions workflow (`.github/workflows/alert-rules-test.yaml`) has been added to execute alert rule tests when triggered manually or by pull requests against specific branches. A new observability rule file (`observability/rules/containeroom.rules.yml`) defines a VMRule for detecting OOM incidents in Kubernetes containers. Additionally, a shell script (`scripts/run_alerts_tests.sh`) automates testing by verifying the PromTool installation, processing alert rule YAMLs, running tests, and cleaning up.

## Changes

| File(s)                                                                                                  | Change Summary                                                                                                                                                                                                                                                                                                                 |
|----------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `.github/workflows/alert-rules-test.yaml`<br>`scripts/run_alerts_tests.sh`                                | Added a workflow and supporting test script to automate PromTool-based alert rule testing. The workflow triggers on manual dispatch and PR events, installs PromTool, and executes the script which handles YAML processing, test execution, and cleanup.                                                                 |
| `observability/rules/containeroom.rules.yml`                                                              | Introduced a new YAML configuration that defines a VMRule for monitoring container OOM events in Kubernetes, including alert settings and metadata annotations.                                                                                                                                                          |

## Sequence Diagram(s)
```mermaid
sequenceDiagram
    participant User as Initiator
    participant GA as GitHub Actions
    participant Checkout as Repository Checkout
    participant Installer as PromTool Installer
    participant Script as run_alerts_tests.sh
    participant Promtool as PromTool

    User->>GA: Trigger workflow (manual dispatch or PR)
    GA->>Checkout: Check out repository using actions/checkout
    GA->>Installer: Install PromTool (fetch, download, extract)
    Installer-->>GA: PromTool installed and version output
    GA->>Script: Execute alert rules test script
    Script->>Promtool: Run tests on alert rule YAMLs
    Promtool-->>Script: Return test results
    Script->>GA: Cleanup temporary files and return status
    GA->>User: Report workflow result
```

## Possibly related PRs
- **FalkorDB/falkordb-dbaas#156**: Introduces a similar GitHub Actions workflow and test script for automating alert rule tests, aligning closely with the changes in this PR.

## Poem
> Hoppin’ through code in fields so wide,  
> I test my rules with joyful pride.  
> A workflow dances and scripts take flight,  
> Ensuring alerts remain just right.  
> With a twitch of whiskers and a bunny cheer,  
> Our code leaps forth—oh, what a year! 🐰✨

<!-- walkthrough_end -->
<!-- internal state start -->


<!-- DwQgtGAEAqAWCWBnSTIEMB26CuAXA9mAOYCmGJATmriQCaQDG+Ats2bgFyQAOFk+AIwBWJBrngA3EsgEBPRvlqU0AgfFwA6NPEgQAfACgjoCEYDEZyAAUASpADK2VmgqyjcEj2wAbb5AokAI7Y0riQ4rjedJAARPgs6FEUuIgxkADuaMiI2ALM6jT0cuGwntiIlJC02LTwAF7wzAKVmPSo8EoY4gBm8NFZkBhOzXwAjACsAOwaMKU8FI0u8tzYFNz4FfzdJai2KMgE6LS0AYjISDmeQ02VowDMABwANOjnGLUM1NHFuHNKiAwFtxxPgsDFegAPSBme4PDRpABiq1+lSUuG03mQASILlqGCIJU8iG4ongvQYjFgmFIyHwfEa3CibC61HgoOQzDQShQWF+qBWvn8QRCiDCLiu+DCvHwEg6dBmHmsdk+WGa6AYDGkFXovwo+GwRFghMg3Xwvnw6Xg+MgAFUbAAZLgAbXizESlBSSoAugAKWC4XDcRAcAD0IaI6lguQ0TGYIYRaG8AGs6QARABCIe6iZTFFoAjA+bQWRDAu8IYmkwAlBojPpjOAoGR6Phtmg8IRSOQqIUFKx2FxePxhKJxFIZPImEoqKp1FodPWTFA4O1kJgcARiGRlL3Y8zOP40OlIDlnK5IMUp8pZ5ptLowIYG6YDBoI79ciH0nSk91vBbECGiYemAFA+NIYA0KKGiyGgzDeBwBgxEhBgWJAACCACSW7dl89Cnpy56tpS1LSEYaGDCQx4AOLqAAErk6FiGyGDIF+FA/n+x69FEgywdEAAGQHJCBYGIBBoTQbB3j8ZAVIyCQZA8rgerVJqtAKggrHfr+Fr7OECxEKQAQtlg/FsRxFoAPq1MS1AMLAMkkFIXRru8/BYGWQrBKEBwuKQ4jWiikD8UoEgya0wWclaMkCFQGD2dILzEqS5KJt48jpKUWD2SRa4BJAnLcoc3HSBkkZWsa/GCBUFASCo8DeOosghqBUQAQAVO14VuVVAg1XVaiNbgzWQSkIadTJ3DULAiAaZ45k6ceVoMN4NSlWgCjxasATxfIFQBlaBK/NQkBEHq2BBv42AsRe8hBQtnG8Ww6BuQE3SUGQmovGlFqHSadKMJgmq+KyoJbDyYDSmdWpXSxtYGIqD26UwLFIJ6REbYgh08UIghPQJo38S8mXwPZsO0ryczeF8oqQFIFBY2DRE2gI124Ngc2QLjAg8ita3ZE5yh+KKJBBlwvQM7gLzqJSohJrSeDGgE6xYwQ57lH9QWCUx7IhglDApngAACEgACzhTrGAzAAciQEJSygYRWqKaXIFYerMNA8R+MU724PZmtUzTYQBFEWSePTjNYN0HvGu7LAkCi5SQDRuD0TzysbOodKyC8tAWhgf5coHnhMBQpzrO8msuAIaUvHbyloExgVzPx0rMAQZoxVaSzfW5zAyn9MuHPxIblBQIZ/p85ZqBg/Gc5HzHg/Hnve3pKJYDZjJoLI8qQAiPe+Lnxrc5AduiHg60noC8DApAU+4egYTzyGAJAmNoEYJZQkpJZo2zYgBy4R8Cw2ND/cIPl4bmEsGhbwNAezMQOCAoKSgVouFBjdIidt1jJGiADFYAhGoUnYOoPoiA6yQFtseJGx4uRKFoFwfiGA+JcGgKEd0yQrptRkhVeeb4owCE/NpTiAEf4iTauJKCME4L8UQshAwEAwBGGqpQAaDUmotVEnrUE6IrSUFdBoVq0hoJwQQkhGIKEYFYS7DuaI+EljgxyviUiBhyLkGPAATTQgAWXtJtXoRBVgYJNA1Twpo+AqNqvVIa8g5IXgUlgK0ylFDYDUnnEgvQMB/Q2gANW8TYMC/0+ADyyWrP6KNdHdggRQfIzCQSbxCMA/geAwCtjAGwAe54fQAHluneKrJtWo9S3iQAANK5EoOQSCnMSonhJAwMkZDjRoSsBhOmlAo4vXoEmK0tBpZV0+AFAkMtUC5PyYU6605Kr4BJD2OkGhZRiDpPAPiylSazVjDJFZGEZgYSdvFVa/wCqJy5NQTGqSjQDCCswtgtlNRbPxuDIKRjObJQWalepzTIh6LXKdc63B8b0H4kmB4s0jEfJ0dobsllXREwyAgMmFSqW4qxviHi4CjGA18AJAAwpSvRFBeneNGQ1KItB56zFQJyt5hkNn0sUkFbBpwo7EwZUaWMs9SqKrqqtYJREmUCqFC7ZIrkdRB1pvA2pwSAhZDBs0P6MQhUiu5XstITlEzYBpk05SIRObgNQCjAJqxoiHCpFIdAjAFjiGnieQWUb5BRCkN4YmkYXoYElBg5A0pZR4gJOCs88gIr/BvsCJex1/l825LQWQMLSb3xUCQTERSwHdHemIaI6x6ARRhdIKamp4aoVgfAzN3q/iiGpgg9k4NsF0l7Pg3IRCz5dFIS4qAVCuGeDoXQRheSClRBkj2olJKyWiRjPy7stKeTBUiWomJmi2raJZAKgx5KTHSTkRYhRz4363w/tdb+SRf7/w0IAsxyFB3WO3D2OxTgCLyH1VSZx5DXEUWPIAxtwsS3P0/gBj0iA/6QMATJOJzRFJJJUqkkNID2wEE5DQY0o0/pERXonUoycf5/VfbMIkWH4kRhupeUoBs/rhONLwaQn1PBES1u3Tu0k+ycnePXCEpDrSWl+C9M+5cAawsQGgUgKBtgnOQOmsIppLm/LCBvdZCxeilTJISfK4oXp3R8ugOqDUVBRAANyGeuX1VR0SNHkpDP/GStQAhPPPKgEgzBgTwb4PnUqpmz4qdFNLMI+pAx4Fxbp/Tm63J23ULNbj1936RpIF6jaEXRw52bTQOLdIHGjQ3ecIzYQksmclIkG11bUto3hn8x2tjaT0zPk3I0XjfEhJ4hVLWN6gvDXvdIcL8BItq2PkwbgsgS5ukOPNgLUTBrBa0f/ZbiBVvrbqxFAetRug7ZbrFyAGtrT8VkIEGShwAiVLmSlOtZ19SXRJjxAIA9ZTWmuuQTUZwHF2e8LQEriof130/gcNzYNKtk3nqNN9MlZkP17HN1uC3jtLZC2Fqoa3avq1ZQSNuHs5N0sbsJx7gxhiVCwdphmS6mBsw+vQaoCxAqhEOjMNC3R4GpYvkcxIfh/4ZcYGHG6F1bpCjByXCBjWqDnnJZTq7rhfltgwK50UyBswNQV1llYnoNrm9WvlPLBmIpFc9OpyFgxQRgDqPok86J2azWgehOBO5EGjs8KgidI6sEQhwXOvgBDF0kPEKuyhlEyu/qOPQxhOGf74eA0Rq9/FkdjVkeYusD4nxNjchjDs2FbH0D3AOQ8aHYMOMvIoa8ahbwLgr0uBT6hLIdHwwEWUlE6CWWNWEesBg++alNqMBgDwGAAAYBAr/n+vkgdwSAACZJgAE5W2jGX+MAAbN0U2O+7gCG3wwU/Dwd880XI2E0owBD7/38WU/aBxgPFoA/n/U2SYU/ffE/B4U/ZfWgBgWgcYNAboSYDUC/cYcYUQe8QwPvZfUYCA2gU2boB4cYU2B4AQJfffEgcA8YffCArfU2O4O4UYUYU2WgU/HfE/AQU2DUZfNAyvfvXAQfBHSyEfPodIcfZsNAoAA== -->

<!-- internal state end -->
<!-- finishing_touch_checkbox_start -->

<details>
<summary>✨ Finishing Touches</summary>

- [ ] <!-- {"checkboxId": "7962f53c-55bc-4827-bfbf-6a18da830691"} --> 📝 Generate Docstrings

</details>

<!-- finishing_touch_checkbox_end -->
<!-- tips_start -->

---



<details>
<summary>🪧 Tips</summary>

### Chat

There are 3 ways to chat with [CodeRabbit](https://coderabbit.ai?utm_source=oss&utm_medium=github&utm_campaign=FalkorDB/falkordb-dbaas&utm_content=157):

- Review comments: Directly reply to a review comment made by CodeRabbit. Example:
  - `I pushed a fix in commit <commit_id>, please review it.`
  - `Generate unit testing code for this file.`
  - `Open a follow-up GitHub issue for this discussion.`
- Files and specific lines of code (under the "Files changed" tab): Tag `@coderabbitai` in a new review comment at the desired location with your query. Examples:
  - `@coderabbitai generate unit testing code for this file.`
  -	`@coderabbitai modularize this function.`
- PR comments: Tag `@coderabbitai` in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
  - `@coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.`
  - `@coderabbitai read src/utils.ts and generate unit testing code.`
  - `@coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.`
  - `@coderabbitai help me debug CodeRabbit configuration file.`

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

### CodeRabbit Commands (Invoked using PR comments)

- `@coderabbitai pause` to pause the reviews on a PR.
- `@coderabbitai resume` to resume the paused reviews.
- `@coderabbitai review` to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
- `@coderabbitai full review` to do a full review from scratch and review all the files again.
- `@coderabbitai summary` to regenerate the summary of the PR.
- `@coderabbitai generate docstrings` to [generate docstrings](https://docs.coderabbit.ai/finishing-touches/docstrings) for this PR.
- `@coderabbitai resolve` resolve all the CodeRabbit review comments.
- `@coderabbitai configuration` to show the current CodeRabbit configuration for the repository.
- `@coderabbitai help` to get help.

### Other keywords and placeholders

- Add `@coderabbitai ignore` anywhere in the PR description to prevent this PR from being reviewed.
- Add `@coderabbitai summary` to generate the high-level summary at a specific location in the PR description.
- Add `@coderabbitai` anywhere in the PR title to generate the title automatically.

### CodeRabbit Configuration File (`.coderabbit.yaml`)

- You can programmatically configure CodeRabbit by adding a `.coderabbit.yaml` file to the root of your repository.
- Please see the [configuration documentation](https://docs.coderabbit.ai/guides/configure-coderabbit) for more information.
- If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: `# yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json`

### Documentation and Community

- Visit our [Documentation](https://docs.coderabbit.ai) for detailed information on how to use CodeRabbit.
- Join our [Discord Community](http://discord.gg/coderabbit) to get help, request features, and share feedback.
- Follow us on [X/Twitter](https://twitter.com/coderabbitai) for updates and announcements.

</details>

<!-- tips_end -->
author_association: CONTRIBUTOR
user: login: github-actions[bot]
created_at: 2025-03-06T11:22:37Z
body: #### Tofu Plan Output - observability_stack_ctrl_plane_k8s
```

[0m[1m[32mNo changes.[0m[1m Your infrastructure matches the configuration.[0m

[0mOpenTofu has compared your real infrastructure against your configuration and
found no differences, so no changes are needed.
```
    
author_association: NONE
user: login: github-actions[bot]
created_at: 2025-03-06T11:22:42Z
body: #### Tofu Plan Output - observability_stack_ctrl_plane_infra
```

[0m[1m[32mNo changes.[0m[1m Your infrastructure matches the configuration.[0m

[0mOpenTofu has compared your real infrastructure against your configuration and
found no differences, so no changes are needed.
```
    
author_association: NONE