Repository: FalkorDB/falkordb-dbaas
Pull request #156: 138 pods ooms are not triggering alerts
Author: login: dudizimber
Created: 2025-03-05T08:28:41Z | Merged: 2025-03-05T08:36:50Z

fix #138 

<!-- This is an auto-generated comment: release notes by coderabbit.ai -->

## Summary by CodeRabbit

- **New Features**
  - Added a new alert that notifies users when Kubernetes containers terminate due to out-of-memory events.
- **Tests**
  - Implemented automated validations to ensure the alert configurations are accurate and reliable.
- **Chores**
  - Streamlined internal workflows to enhance testing processes for alert rules.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Linked issue #138: Pods OOMs are not triggering alerts

Diff:
diff --git a/.github/workflows/alert-rules-test.yaml b/.github/workflows/alert-rules-test.yaml
new file mode 100644
index 000000000..470f0529f
--- /dev/null
+++ b/.github/workflows/alert-rules-test.yaml
@@ -0,0 +1,33 @@
+name: Test alert rules
+on:
+  workflow_dispatch:
+  pull_request:
+    branches:
+      - dev
+      - main
+    paths:
+      - 'observability/rules/**'
+      - 'observability/tests/**'
+
+concurrency:
+  group: alert-rules-${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+      
+      - name: Install PromTool
+        run: |
+          VERSION=$(curl -Ls https://api.github.com/repos/prometheus/prometheus/releases/latest | jq ".tag_name" | xargs | cut -c2-)
+          wget -qO- "https://github.com/prometheus/prometheus/releases/download/v${VERSION}/prometheus-$VERSION.linux-amd64.tar.gz" \
+            | tar xvzf - "prometheus-$VERSION.linux-amd64"/promtool --strip-components=1
+          sudo mv promtool /usr/local/bin/promtool
+          promtool --version
+
+      - name: Run tests
+        run: ./scripts/run_alerts_tests.sh
+
diff --git a/observability/rules/containeroom.rules.yml b/observability/rules/containeroom.rules.yml
new file mode 100644
index 000000000..996563fb6
--- /dev/null
+++ b/observability/rules/containeroom.rules.yml
@@ -0,0 +1,17 @@
+apiVersion: operator.victoriametrics.com/v1beta1
+kind: VMRule
+metadata:
+  namespace: observability
+  name: k8s.rules.containeroom
+spec:
+  groups:
+    - name: k8s.rules.container_oom
+      rules:
+        - alert: ContainerOOMKilled
+          expr: kube_pod_container_status_last_terminated_reason{reason="OOMKilled"} == 1
+          for: 0m
+          labels:
+            severity: critical
+          annotations:
+            summary: "Container OOMKilled"
+            description: "Container {{ $labels.pod }} ({{ $labels.container }}) in namespace {{ $labels.namespace }} was terminated due to OOMKilled."
diff --git a/observability/rules/tests/containeroom.rules.test.yml b/observability/rules/tests/containeroom.rules.test.yml
new file mode 100644
index 000000000..dba284793
--- /dev/null
+++ b/observability/rules/tests/containeroom.rules.test.yml
@@ -0,0 +1,24 @@
+rule_files:
+  - "./rules/containeroom.rules.yml"
+
+evaluation_interval: 10m
+
+tests:
+  - interval: 1m
+    input_series:
+      - series: 'kube_pod_container_status_last_terminated_reason{reason="OOMKilled", pod="test-pod", namespace="test-ns", container="test-container"}'
+        values: "1 1 1 1 1"
+
+    alert_rule_test:
+      - eval_time: 1m
+        alertname: ContainerOOMKilled
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              pod: "test-pod"
+              namespace: "test-ns"
+              container: "test-container"
+              reason: "OOMKilled"
+            exp_annotations:
+              summary: "Container OOMKilled"
+              description: "Container test-pod (test-container) in namespace test-ns was terminated due to OOMKilled."
diff --git a/scripts/run_alerts_tests.sh b/scripts/run_alerts_tests.sh
new file mode 100755
index 000000000..6d7d69b26
--- /dev/null
+++ b/scripts/run_alerts_tests.sh
@@ -0,0 +1,35 @@
+#!/bin/bash
+
+# Check that promtool is installed
+if ! command -v promtool &> /dev/null
+then
+    echo "promtool could not be found"
+    exit 1
+fi
+
+mkdir -p observability/rules/tests/rules
+
+# For each file under observability/rules directory, create a temp yaml file and run the tests under observability/rules/tests directory
+for file in observability/rules/*.yml
+do
+  # Create temp rule file under observability/rules/tests directory
+  cp $file observability/rules/tests/rules/
+  yq eval -i '.groups = .spec.groups | del(.apiVersion, .kind, .metadata, .groups[].params, .spec)' observability/rules/tests/rules/$(basename $file)
+done
+
+errors=0
+for test_file in observability/rules/tests/*.test.yml
+do
+  promtool test rules $test_file
+  if [ $? -ne 0 ]; then
+    errors=$((errors+1))
+  fi
+done
+
+# Clean up temp files
+rm -rf observability/rules/tests/rules
+
+if [ $errors -ne 0 ]; then
+  echo "Tests failed"
+  exit 1
+fi
\ No newline at end of file


Reviews:
user: login: gemini-code-assist[bot]
body: Hello @dudizimber, I'm Gemini Code Assist[^1]! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

## Summary of Changes
This pull request addresses issue #138, which involves ensuring that alerts are triggered when pods experience OOM (Out Of Memory) kills. I've added a new alert rule to detect this condition and a corresponding test to verify its functionality. Additionally, I've set up a GitHub workflow to automatically run these alert rule tests on pull requests.

### Highlights
* **Alert Rule**: A new alert rule `ContainerOOMKilled` has been added to `observability/rules/containeroom.rules.yml`. This rule triggers when a container is terminated due to OOMKilled.
* **Testing**: A test case has been added in `observability/rules/tests/containeroom.rules.test.yml` to verify the `ContainerOOMKilled` alert rule.
* **CI Workflow**: A new GitHub workflow `alert-rules-test.yaml` has been created to automatically run alert rule tests on pull requests that modify files in `observability/rules/**` or `observability/tests/**`.
* **Testing Script**: The `run_alerts_tests.sh` script has been updated to handle VMRule resources correctly by extracting the rule groups from the spec.

### Changelog
<details>
<summary>Click here to see the changelog</summary>

* **.github/workflows/alert-rules-test.yaml**
  * Added a new workflow to run alert rule tests on pull requests.
  * The workflow checks out the repository, installs Promtool, and executes the `run_alerts_tests.sh` script.
* **observability/rules/containeroom.rules.yml**
  * Created a new VMRule resource to detect container OOMKilled events.
  * The rule triggers an alert named `ContainerOOMKilled` with severity `critical`.
* **observability/rules/tests/containeroom.rules.test.yml**
  * Added a test case for the `ContainerOOMKilled` alert rule.
  * The test verifies that the alert is triggered when the `kube_pod_container_status_last_terminated_reason` metric indicates an OOMKilled event.
* **scripts/run_alerts_tests.sh**
  * Added logic to copy the rule file to a temporary location.
  * Added logic to extract the rule groups from the spec of the VMRule resource before running the tests.
  * Added logic to clean up the temporary files after the tests are complete.

</details>


<details>

<summary><b>Using Gemini Code Assist</b></summary>
<br>

The full guide for Gemini Code Assist can be found on our [documentation page](https://developers.google.com/gemini-code-assist/docs/review-github-code), here are some quick tips.

<b>Invoking Gemini</b>

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either `/gemini <command>` or `@gemini-code-assist <command>`. Below is a summary of the supported commands.

Feature | Command | Description
--- | --- | ---
Code Review | `/gemini review` | Performs a code review for the current pull request in its current state.
Pull Request Summary | `/gemini summary` | Provides a summary of the current pull request in its current state.
Comment | @gemini-code-assist | Responds in comments when explicitly tagged, both in issue comments and review comments.
Help | `/gemini help` | Displays a list of available commands.

<b>Customization</b>

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a `.gemini/` folder in the base of the repository. Detailed instructions can be found [here](https://developers.google.com/gemini-code-assist/docs/review-github-code).

<b>Limitations & Feedback</b>

Gemini Code Assist is currently in preview and may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with :thumbsup: and :thumbsdown: on @gemini-code-assist comments to provide feedback.

</details>

<b>You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the [Gemini Code Assist IDE Extension](https://cloud.google.com/products/gemini/code-assist).</b>


[^1]: Review the [Privacy Notices](https://policies.google.com/privacy), [Generative AI Prohibited Use Policy](https://policies.google.com/terms/generative-ai/use-policy), [Terms of Service](https://policies.google.com/terms), and learn how to configure Gemini Code Assist in GitHub [here](https://developers.google.com/gemini-code-assist/docs/review-github-code). Gemini can make mistakes, so double check it and [use code with caution](https://support.google.com/legal/answer/13505487).

---

**Trivia time!**

What does OOM stand for in the context of computer science?

<details>
<summary><b>Click here for the answer</b></summary>
<br>
OOM stands for Out Of Memory.

</details>

state: COMMENTED
author_association: NONE
submitted_at: 2025-03-05T08:28:50Z
user: login: gemini-code-assist[bot]
body: ## Code Review
The pull request introduces a new alert rule for detecting Container OOMKilled events and adds a corresponding test. The changes are well-structured and the alert seems reasonable. The workflow setup for running alert rule tests is also a good addition. However, there are a few minor points that could be improved.

### Summary of Findings
* **Promtool version**: The workflow installs a specific version of Promtool (v3.2.0). It's generally better to use the latest version or a more recent one to benefit from bug fixes and new features. Consider using a more dynamic approach to ensure the workflow uses an up-to-date version.
* **Temporary file cleanup in shell script**: The shell script creates temporary files under `observability/rules/tests/rules/`. While the script attempts to clean up these files with `rm -rf observability/rules/tests/rules`, there's a potential race condition if the script exits prematurely (e.g., due to an error in `promtool test rules`). Consider using `trap` to ensure the cleanup happens even if the script exits early.
* **Missing labels in alert rule test**: The alert rule test in `containeroom.rules.test.yml` includes `reason: "OOMKilled"` in the `exp_labels` section. This label is not present in the alert definition in `containeroom.rules.yml`. While the test will still pass, it's misleading and could cause confusion. The `reason` label should be removed from the test or added to the alert definition.

### Merge Readiness
The pull request is almost ready for merging. The new alert rule and tests are a valuable addition. However, I recommend addressing the issues related to the Promtool version, temporary file cleanup, and the missing label in the alert rule test before merging. I am unable to approve this pull request, and recommend that others review and approve this code before merging.
state: CHANGES_REQUESTED
author_association: NONE
submitted_at: 2025-03-05T08:29:05Z
user: login: coderabbitai[bot]
body: **Actionable comments posted: 1**

<details>
<summary>🧹 Nitpick comments (2)</summary><blockquote>

<details>
<summary>.github/workflows/alert-rules-test.yaml (2)</summary><blockquote>

`22-22`: **Remove Trailing Spaces**

There is a trailing space on line 22. Please remove any trailing whitespace to maintain clean YAML formatting.

<details>
<summary>🧰 Tools</summary>

<details>
<summary>🪛 YAMLlint (1.35.1)</summary>

[error] 22-22: trailing spaces

(trailing-spaces)

</details>

</details>

---

`33-33`: **Eliminate Extra Blank Line**

Line 33 contains an extra blank line. Removing extra blank lines enhances readability and consistency with standard YAML practices.

<details>
<summary>🧰 Tools</summary>

<details>
<summary>🪛 YAMLlint (1.35.1)</summary>

[warning] 33-33: too many blank lines

(1 > 0) (empty-lines)

</details>

</details>

</blockquote></details>

</blockquote></details>

<details>
<summary>📜 Review details</summary>

**Configuration used: CodeRabbit UI**
**Review profile: CHILL**
**Plan: Pro**

<details>
<summary>📥 Commits</summary>

Reviewing files that changed from the base of the PR and between 65ebab2302ce456f08cff9972a4fc662c2ebda1c and f48bfc4ec4793ea4e037c3f063b7e3d4b793c09c.

</details>

<details>
<summary>📒 Files selected for processing (4)</summary>

* `.github/workflows/alert-rules-test.yaml` (1 hunks)
* `observability/rules/containeroom.rules.yml` (1 hunks)
* `observability/rules/tests/containeroom.rules.test.yml` (1 hunks)
* `scripts/run_alerts_tests.sh` (1 hunks)

</details>

<details>
<summary>🧰 Additional context used</summary>

<details>
<summary>🪛 Shellcheck (0.10.0)</summary>

<details>
<summary>scripts/run_alerts_tests.sh</summary>

[warning] 17-17: Quote this to prevent word splitting.

(SC2046)

</details>

</details>
<details>
<summary>🪛 YAMLlint (1.35.1)</summary>

<details>
<summary>.github/workflows/alert-rules-test.yaml</summary>

[error] 22-22: trailing spaces

(trailing-spaces)

---

[warning] 33-33: too many blank lines

(1 > 0) (empty-lines)

</details>

</details>

</details>

<details>
<summary>⏰ Context from checks skipped due to timeout of 90000ms (1)</summary>

* GitHub Check: Analyze (javascript-typescript)

</details>

<details>
<summary>🔇 Additional comments (4)</summary><blockquote>

<details>
<summary>observability/rules/containeroom.rules.yml (1)</summary>

`1-17`: **Clear and Well-Structured Alert Rule**

The YAML configuration for the `ContainerOOMKilled` alert is clearly defined. The API version, kind, metadata, and spec are all structured as expected. The expression  
```
kube_pod_container_status_last_terminated_reason{reason="OOMKilled"} == 1
```  
accurately reflects the intent of triggering an alert when a container is killed due to an OOM event. Please verify that the immediate trigger (`for: 0m`) is intentional; if a brief delay might help filter transient conditions, consider adjusting it in the future.

</details>
<details>
<summary>observability/rules/tests/containeroom.rules.test.yml (1)</summary>

`1-25`: **Comprehensive Alert Rule Test Configuration**

The test configuration nicely references the alert rule file and defines a clear test case—including the input series and all expected alert labels and annotations. This level of detail will help ensure that any future changes to alert rules are properly validated.

</details>
<details>
<summary>scripts/run_alerts_tests.sh (1)</summary>

`1-35`: **Overall Script Functionality**

The script is well-designed to:
- Validate the existence of `promtool`.
- Create a temporary directory for rule processing.
- Iterate over alert rule files, process them using `yq`, run the tests via `promtool`, and clean up afterward.

Ensure that the paths used (e.g., `observability/rules`, `observability/rules/tests/rules`) are correct within your repository structure and that you have the necessary permissions. The error handling for missing `promtool` and tracking of test errors is appropriate.

<details>
<summary>🧰 Tools</summary>

<details>
<summary>🪛 Shellcheck (0.10.0)</summary>

[warning] 17-17: Quote this to prevent word splitting.

(SC2046)

</details>

</details>

</details>
<details>
<summary>.github/workflows/alert-rules-test.yaml (1)</summary>

`1-34`: **Well-Structured GitHub Actions Workflow**

The workflow correctly triggers on `workflow_dispatch` and pull requests affecting the alert rules and tests. It clearly defines steps to check out the repository, install PromTool, and run the alert tests via the provided script. Verify that PromTool version 3.2.0 meets your requirements and that the overall setup fits your CI/CD process.

<details>
<summary>🧰 Tools</summary>

<details>
<summary>🪛 YAMLlint (1.35.1)</summary>

[error] 22-22: trailing spaces

(trailing-spaces)

---

[warning] 33-33: too many blank lines

(1 > 0) (empty-lines)

</details>

</details>

</details>

</blockquote></details>

</details>

<!-- This is an auto-generated comment by CodeRabbit for review status -->
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-05T08:31:07Z

Review comments:
diff_hunk: @@ -0,0 +1,35 @@
+#!/bin/bash
+
+# Check that promtool is installed
+if ! command -v promtool &> /dev/null
+then
+    echo "promtool could not be found"
+    exit 1
+fi
+
+mkdir -p observability/rules/tests/rules
+
+# For each file under observability/rules directory, create a temp yaml file and run the tests under observability/rules/tests directory
+for file in observability/rules/*.yml
+do
+  # Create temp rule file under observability/rules/tests directory
+  cp $file observability/rules/tests/rules/
+  yq eval -i '.groups = .spec.groups | del(.apiVersion, .kind, .metadata, .groups[].params, .spec)' observability/rules/tests/rules/$(basename $file)
+done
+
+errors=0
+for test_file in observability/rules/tests/*.test.yml
+do
+  promtool test rules $test_file
+  if [ $? -ne 0 ]; then
+    errors=$((errors+1))
+  fi
+done
+
+# Clean up temp files
+rm -rf observability/rules/tests/rules
path: scripts/run_alerts_tests.sh
user: login: gemini-code-assist[bot]
body: ![high](https://img.shields.io/badge/priority-high-orange)

Consider using `trap` to ensure the temporary files are cleaned up even if the script exits prematurely. This will prevent leaving temporary files behind in case of an error.

```suggestion
trap 'rm -rf observability/rules/tests/rules' EXIT
```
created_at: 2025-03-05T08:29:05Z
author_association: NONE
diff_hunk: @@ -0,0 +1,33 @@
+name: Run tests for alert rules
+on:
+  workflow_dispatch:
+  pull_request:
+    branches:
+      - dev
+      - main
+    paths:
+      - 'observability/rules/**'
+      - 'observability/tests/**'
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+      
+      - name: Install PromTool
+        run: |
+          curl -O https://github.com/prometheus/prometheus/releases/download/v3.2.0/prometheus-3.2.0.linux-amd64.tar.gz
+          tar -xvf prometheus-3.2.0.linux-amd64.tar.gz
+          sudo cp prometheus-3.2.0.linux-amd64/promtool /usr/local/bin/
+          chmod +x /usr/local/bin/promtool
+          rm -rf prometheus*
path: .github/workflows/alert-rules-test.yaml
user: login: gemini-code-assist[bot]
body: ![medium](https://img.shields.io/badge/priority-medium-yellow)

Consider using a more recent version of Promtool, or even better, use `latest` to always get the newest version. Also, consider using `wget` instead of `curl` as it's more commonly available on minimal systems. You can also simplify the installation by directly extracting the binary and setting execute permissions.

```
      - name: Install PromTool
        run: |
          wget https://github.com/prometheus/prometheus/releases/download/v3.2.0/prometheus-3.2.0.linux-amd64.tar.gz
          tar -xvf prometheus-3.2.0.linux-amd64.tar.gz
          sudo cp prometheus-3.2.0.linux-amd64/promtool /usr/local/bin/
          sudo chmod +x /usr/local/bin/promtool
          rm -rf prometheus*
```
created_at: 2025-03-05T08:29:05Z
author_association: NONE
diff_hunk: @@ -0,0 +1,24 @@
+rule_files:
+  - "./rules/containeroom.rules.yml"
+
+evaluation_interval: 10m
+
+tests:
+  - interval: 1m
+    input_series:
+      - series: 'kube_pod_container_status_last_terminated_reason{reason="OOMKilled", pod="test-pod", namespace="test-ns", container="test-container"}'
+        values: "1 1 1 1 1"
+
+    alert_rule_test:
+      - eval_time: 1m
+        alertname: ContainerOOMKilled
+        exp_alerts:
+          - exp_labels:
+              severity: critical
+              pod: "test-pod"
+              namespace: "test-ns"
+              container: "test-container"
+              reason: "OOMKilled"
path: observability/rules/tests/containeroom.rules.test.yml
user: login: gemini-code-assist[bot]
body: ![medium](https://img.shields.io/badge/priority-medium-yellow)

The `reason` label is not present in the alert definition in `containeroom.rules.yml`. It's best to keep the test aligned with the actual alert definition. Should this be removed?

```
              container: "test-container"
```
created_at: 2025-03-05T08:29:05Z
author_association: NONE
diff_hunk: @@ -0,0 +1,35 @@
+#!/bin/bash
+
+# Check that promtool is installed
+if ! command -v promtool &> /dev/null
+then
+    echo "promtool could not be found"
+    exit 1
+fi
+
+mkdir -p observability/rules/tests/rules
+
+# For each file under observability/rules directory, create a temp yaml file and run the tests under observability/rules/tests directory
+for file in observability/rules/*.yml
+do
+  # Create temp rule file under observability/rules/tests directory
+  cp $file observability/rules/tests/rules/
+  yq eval -i '.groups = .spec.groups | del(.apiVersion, .kind, .metadata, .groups[].params, .spec)' observability/rules/tests/rules/$(basename $file)
path: scripts/run_alerts_tests.sh
user: login: gemini-code-assist[bot]
body: ![medium](https://img.shields.io/badge/priority-medium-yellow)

The `yq` command modifies the file in place. If the `yq` command fails, the original file will be corrupted. Consider creating a copy of the file before modifying it, and only replacing the original file if the `yq` command succeeds.

```suggestion
  cp $file observability/rules/tests/rules/
  yq eval -i '.groups = .spec.groups | del(.apiVersion, .kind, .metadata, .groups[].params, .spec)' observability/rules/tests/rules/$(basename $file)
```
created_at: 2025-03-05T08:29:05Z
author_association: NONE
diff_hunk: @@ -0,0 +1,35 @@
+#!/bin/bash
+
+# Check that promtool is installed
+if ! command -v promtool &> /dev/null
+then
+    echo "promtool could not be found"
+    exit 1
+fi
+
+mkdir -p observability/rules/tests/rules
+
+# For each file under observability/rules directory, create a temp yaml file and run the tests under observability/rules/tests directory
+for file in observability/rules/*.yml
+do
+  # Create temp rule file under observability/rules/tests directory
+  cp $file observability/rules/tests/rules/
+  yq eval -i '.groups = .spec.groups | del(.apiVersion, .kind, .metadata, .groups[].params, .spec)' observability/rules/tests/rules/$(basename $file)
path: scripts/run_alerts_tests.sh
user: login: coderabbitai[bot]
body: _⚠️ Potential issue_

**Quote Variable Expansions to Prevent Word Splitting**

The command on line 17 uses variable expansion without quotes, which could lead to unexpected word splitting in certain cases. Please update the command as follows:

```diff
-  yq eval -i '.groups = .spec.groups | del(.apiVersion, .kind, .metadata, .groups[].params, .spec)' observability/rules/tests/rules/$(basename $file)
+  yq eval -i '.groups = .spec.groups | del(.apiVersion, .kind, .metadata, .groups[].params, .spec)' observability/rules/tests/rules/$(basename "$file")
```

This change ensures a robust handling of file names, especially those containing spaces.




<!-- suggestion_start -->

<details>
<summary>📝 Committable suggestion</summary>

> ‼️ **IMPORTANT**
> Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

`````suggestion
  cp $file observability/rules/tests/rules/
  yq eval -i '.groups = .spec.groups | del(.apiVersion, .kind, .metadata, .groups[].params, .spec)' observability/rules/tests/rules/$(basename "$file")
`````

</details>

<!-- suggestion_end -->

<details>
<summary>🧰 Tools</summary>

<details>
<summary>🪛 Shellcheck (0.10.0)</summary>

[warning] 17-17: Quote this to prevent word splitting.

(SC2046)

</details>

</details>

<!-- This is an auto-generated comment by CodeRabbit -->
created_at: 2025-03-05T08:31:06Z
author_association: CONTRIBUTOR

PR comments:
user: login: coderabbitai[bot]
created_at: 2025-03-05T08:28:48Z
body: <!-- This is an auto-generated comment: summarize by coderabbit.ai -->
<!-- This is an auto-generated comment: skip review by coderabbit.ai -->

> [!IMPORTANT]
> ## Review skipped
> 
> Auto reviews are disabled on base/target branches other than the default branch.
> 
> 
> 
> Please check the settings in the CodeRabbit UI or the `.coderabbit.yaml` file in this repository. To trigger a single review, invoke the `@coderabbitai review` command.
> 
> You can disable this status message by setting the `reviews.review_status` to `false` in the CodeRabbit configuration file.

<!-- end of auto-generated comment: skip review by coderabbit.ai -->
<!-- walkthrough_start -->

## Walkthrough
A new GitHub Actions workflow has been introduced to automate testing for alert rules. This workflow is triggered by manual dispatches and pull requests on the dev and main branches, with concurrency settings that cancel previous runs when new ones are initiated. Additionally, a new observability rule monitors container terminations due to out-of-memory events and is accompanied by a test configuration. A shell script has been added to execute these tests, processing YAML rule files with PromTool and cleaning up temporary files afterward.

## Changes
| File(s)                                                                                                 | Summary |
|---------------------------------------------------------------------------------------------------------|---------|
| `.github/.../alert-rules-test.yaml`                                                                     | New GitHub Actions workflow for testing alert rules on workflow_dispatch and pull_request events (branches: dev and main) with concurrency controls. |
| `observability/rules/containeroom.rules.yml`<br>`observability/rules/tests/containeroom.rules.test.yml`    | New YAML files: one defines the ContainerOOMKilled alert rule for monitoring OOM conditions; the other provides test configuration with evaluation intervals and expected outputs. |
| `scripts/run_alerts_tests.sh`                                                                           | New shell script that validates promtool availability, processes YAML rule files (using yq), runs tests with promtool, cleans up test files, and exits with appropriate status. |

## Sequence Diagram(s)
```mermaid
sequenceDiagram
    participant E as GitHub Event
    participant W as Workflow
    participant J as Test Job
    participant S as Test Script
    participant PT as PromTool

    E->>W: Trigger (workflow_dispatch or pull_request)
    W->>J: Start test job on ubuntu-latest
    J->>J: Checkout repository and install PromTool
    J->>S: Execute run_alerts_tests.sh
    S->>PT: Check for PromTool availability
    S->>S: Process YAML files for alert rules
    S->>PT: Run tests on updated YAML files
    S->>J: Report and return test results
```

## Poem
> I'm a rabbit dancing through code so light,  
> With workflows and tests that shine so bright.  
> YAML rules and scripts in a merry parade,  
> Alerting clear paths where errors fade.  
> Hopping forward, our fixes lead the way! 🐇✨

<!-- walkthrough_end -->
<!-- internal state start -->


<!-- DwQgtGAEAqAWCWBnSTIEMB26CuAXA9mAOYCmGJATmriQCaQDG+Ats2bgFyQAOFk+AIwBWJBrngA3EsgEBPRvlqU0AgfFwA6NPEgQAfACgjoCEYDEZyAAUASpADK2VmgqyjcEj2wAbb5AokAI7Y0riQ4rjedJAARACMAMwAHDyKyPgsyC6eGPhhuBTwRKSFGEToURS4iDGQAO5oyIjYAszqNPRy4bCe2IiUkLTYtPAAXvDMAgOY9KjwShjiAGbw0V24PZAYTlN8cQCsAGwaMJu8Ey7y3NgU3Pj9/EvdqLYoyATotLQBiMhIzTkdgNEkkADT1BAMWA8Si4bQYd74bqeJbabw3Tz4J74PBgLFgNjMfCuSAACgA8uSALIASkgJCkiz+WBBqVoiPChWK00q1ROHkG0gYhW44nwWCxyOsdngGAY6KUWX8JCWlDIDE8HxWAA9ZeUNqhENxRPAVgw3gCTgAxG4bAZKOHwbzIAJEFwjMpSibcKJsRbUeDi/h8KGYUjIZhoJQoLB2hTR7JbPI8Cj4CTzaKyqXNZyufmbV4MTCQKboBga37RDap7BEaFxpb4Xz4Op6yAAVRsABkuABtWC4XDcRAcAD0o6I6lgLQ0TGYo6taG8AGtiQARABCo9RK+JtAEYH3aEao+uvlHB0OAF1SQOhyPx5ONjO5wul6uKJvt++9wejyez28C8jhpDQjH0YxwCgMh6ElNA8EIUhyCoDoFFYdguF4fhhFEcQpBkeQmCUKhVHULQdAgkwoDgOYsiweCCGIMhlFQuc/U4fw0DqSAc0jEkuiI5RSM0bRdDAQxINMAwNCfacBFHOpiWXJZvBbRBRyXWEwAoHxpDAGhEE0WQ0GYbwOAMGJLIMCxIAAQQASSY5DqGiXjLkeRhYDDaQjFsrYSG4gBxdQAAkWjssRAwReolJUltIBWKIthM6IAANNKqbTdMQfTQg0YzTNSyAvJkEgyBjApFGwDV6A+BiWBcqUDPET04N5fxsvzTxFIoZTVO41ACiKEpoiDXBFPpRlqi4VKer6lsAH0RiNagoSKmZIFSwCFoCYJQlS8EjRNM0l28eRGz4AQqDlHpkFSpQJHWjB6FSyNZQO9BnpQMIiQwdRiWQUMymkGMpW4ahYDuwR+goCQVCddRZFHHSonUgAqNGnpe6HKDhtRvER0dmvRzGuoUOUbgCOV5H6Qc9WQMhmgCbpqH4DAzrZzxZUMzANQ8uM5ri7idOi1m0HCCYUWJdBIEnRkS2uqFwSLOUSAJz1MHkWUwF4fAiB+F1sGi00ZfIbjxS594uRG2gwIMAVBf68nECQaoPPFl2yiSoRBGStgXuaj66khaERfSWNNm8FzDMgKQKBdoNJXbAQjdwbAyZ9gQY3lYYQf6OOlx4mhhy4FZ49wcF1E80Rl3SPApQCO4XYIEk+jbON0si8V1KhGucVwAABCQABZ1q7jATgAORIbUK++mMed8ZArFTZhoAyPwuloFt2fwKN282I6GFNVZ6DjhOJSeFeWBIO0+nBGeCjQSLPXUcENqYbhZAPzwttXggmxFTUBgdyHxUqjj6BQUcqkixAWAaOD6jMbhtirqgGeog8AqCiNaWUp1ZCVzCOghgeAQYe2FPAUUkAYEuXoKzVKGhRyIHIaKdSIsFoZWqAtYmGhECwCKh8EWTVQiIDtuYSwtlvA0BQlFDkcYlDyhcAGbuHkZ53CqKNPg1wBAE3NOwdQqxEDgUgNPbijt4pRiULQGaNgjbhGEQlaWHCOqoyKlmehskWgKViv1dSHCsqo1yoZfKJlvCpQslZAwEAwBGBxrDeGBNcBIxRtIUcTB/SykoBkZgGhkkiNkKZcylkYjWXEY5JCLFXJOD4vISUQNwy+X8txAAmrZKkXZyYrCIDcJRWBEpSz4LEvGCNEnFUaCWMqWBZSVSGDVcESgVh/Q1pAAAalSGxSULqQF+v9Uo5Q0mOmQnYigbQQFiiwEMTUSJ+54iWASEgRISQUmpHSNJIwznMkgAAaRaJQcgBkyZ9J4saY+KwQZxlslYeysdKAX0+vQZcspaCV2evAIsLVyioKVKs9ZvRnoDA7vgY0KFiQaHTGIYk8AUpDQYCIucRUIX2ROPZMIsoc6Ki2bfKM1APbVWhGMuMIC2ArT5htQVmInhxmSWTI+J80VRX4HgdWpDZY1m4H7NKy4kgiNybOcUBzKALSyUHEO5MDlKk9kQJKTjknqpegAYT1fCSglIqSfKdFEWg9DTioBtUNbk8cITlTjKog28rUrLh+QtO4tAFr7KdRQBaPM06IAWlHQyXDKAnJoTtEgjRxQAG8Ah5owAAXhiC6t1vg6AxAAL6QBLSWyAcQioMiXNgaO4QkQFBCMikYcrPQbDFqa+N9R+WZtwahC5nbPqQBdeTN5UUyZONQGkzpGJapIi8lIGW5DxCwJ4gySgiMqGHu8OCVsGxPq5DhO8lMaZ5hth5bmeQG0HRomiIqZhZz+abEPYsZFbKf7oCWKqMQ0Ro1wuHRkvgYqsjIDqGrPw/LNiwfBhqURJS7KSJYjI6dcjRBR2kcoyUqjiSoWllonR9JFj6J8pE4xAVnGeAsXQGaDr0nIQre6ugrisCpUGfEwmuTUmOug1knJnV8mhPCcUyJUkBP4yE9lImwiRMccySwCTqMNDNXygUmTmGHJOQqfQNyJJaleWBoYgwfkzaQBaW0jpRRunfsBZs5qj72q5NGaVcqUzUwzLoEysIMrQXvE2DavpHJSytvRDQ89U4ZYBFVFTPmYC41ic0zqqTH0Nq0zopNNtPSKq40LpKOIAAGLZsoSEiNOCiJ0XM5QKmVc1RgjRNReTCGHSalB5AnJIeCPACNxga0mRga4IWj1gq6zxCYPgO3iwy4cpNfQOXUpjH2gMY3YhcarbQWoRbEBBk2R7YFsq2SQeWwMC9CB6KChdkQU50RUPPxIEurzuk7Ex3mRkhm2pgWoQ4YgADCogNOLFbEdj+qKB7Y9aCWoeXzthfpAD3C0Qo5TGdJBzA16enIHc5sDhZNVHo/oJjtWzI2WeHzkexJ4Jo3glexqZWonkLvy+kd8U56EBJTjLjvI+O73pgTDxKp7lX1ChFN+yU/P2pMLIC4QMGGbISKkULj4+GFFEeiiRgHZGNFeG0ai6jEQDFGJMd9sIq7nM68+JY0G/GBAwyGQkpJyniZqZh+JnVumcv1ESw2Js/U2yGR0mIDE5lIC6E2skhaUXUpcC7K7KDizyipRiAw4T13UxZck6ZGIYTo9QFSrF9tZyFpTNK6Erg9hb7TtSpV5gReY+pWJonyA0P4RKja0WB4t3NpV7iaE8EqVZSTcTdNxAuWvrpV5DtXSGbDJFV+39d5YEingSkkw6XrCjbsN5Cm7hvDClWVV2U5iKFKnPo8nUujtnGO8MQzxL9sf9/A6X3yXhRUSrjL84sALaqKsJEeqSMGgIRQyNsNqWENsbzVfTMLABzdpKLAPA0COX+BTYZd3FxQYeAAIclPMerF/aXcZScaKASHoBgBFT0QnTwXgaQdUcVKUP+FgABUJNCSMZ6B+XUdFVAvlLASgVMPgIVRANAUgFAJ4TFJMMIRsI2W2SAZlZELAYUXNAyGWEYfAluc6aWNrbzE2KubeEGa9CoItWgeQGeV2aVV/dQCpdIOOekZ+aEJAhKRrUGAlZ3UrRTRJZGbKFfPA3CYkfBBQL+H+ZgPDTYJ3F3QTbw4TT3XJPwzQwIyDIkEYJYb+AdHoPA01dgSANuT0VKWQQIfhEA2gIQPofIQ+btCPZmLoAIA5KA9meQOMfWHEYcSDAIIkdMT0I2cgCsUQkkUFbwdkO2RwDwvaRYM6cEOMHfChbrI2d4exIMXNKELZRQE+aIZwwFahVCLMdwqIrw7AlJdvXAxI1uC1TaXWZgNgr1RQp+Kg8LQESYAYEjCgIQhmOUHERYNUegIYXZCA3g3WPok4WyJYKRCoPwYmdAZmIhEhJFbMV/eUXNaKbANVWo+5e9DIzUe5NRUBUIFw7TBQp4TWK3AnNEAhBVIcPAJUVEJ0DEDlX4MQ5jL6Cwt2AfcWXIDAMAUYTJIuagNbQSFXcRbDHXWRTYeRQjIXPXNRcjTRFoKjPRcQOjKAS3GYyhFjKxN/DAA/WEI/YRHhPhR3VU6oBBAzOTCSKiajWCIkhCYzK/egNiDCTibiMzeQASRQISNQESCicSAwC0tidQSvdkHNdMAKOgRNOEKoMSc0qCBKYeJIAQJYBgYeUQYeAAdgAE4Ehc1kyKsEhUyGAEglgKtDgEgBBUySAEhaBh4yzMyGAKt0zzQIJfSYz/TcBAyU0AgQyEMY0YIoygA=== -->

<!-- internal state end -->
<!-- tips_start -->

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

<details>
<summary>❤️ Share</summary>

- [X](https://twitter.com/intent/tweet?text=I%20just%20used%20%40coderabbitai%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20the%20proprietary%20code.%20Check%20it%20out%3A&url=https%3A//coderabbit.ai)
- [Mastodon](https://mastodon.social/share?text=I%20just%20used%20%40coderabbitai%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20the%20proprietary%20code.%20Check%20it%20out%3A%20https%3A%2F%2Fcoderabbit.ai)
- [Reddit](https://www.reddit.com/submit?title=Great%20tool%20for%20code%20review%20-%20CodeRabbit&text=I%20just%20used%20CodeRabbit%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20proprietary%20code.%20Check%20it%20out%3A%20https%3A//coderabbit.ai)
- [LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fcoderabbit.ai&mini=true&title=Great%20tool%20for%20code%20review%20-%20CodeRabbit&summary=I%20just%20used%20CodeRabbit%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20proprietary%20code)

</details>

<details>
<summary>🪧 Tips</summary>

### Chat

There are 3 ways to chat with [CodeRabbit](https://coderabbit.ai?utm_source=oss&utm_medium=github&utm_campaign=FalkorDB/falkordb-dbaas&utm_content=156):

- Review comments: Directly reply to a review comment made by CodeRabbit. Example:
  - `I pushed a fix in commit <commit_id>, please review it.`
  - `Generate unit testing code for this file.`
  - `Open a follow-up GitHub issue for this discussion.`
- Files and specific lines of code (under the "Files changed" tab): Tag `@coderabbitai` in a new review comment at the desired location with your query. Examples:
  - `@coderabbitai generate unit testing code for this file.`
  -	`@coderabbitai modularize this function.`
- PR comments: Tag `@coderabbitai` in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
  - `@coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.`
  - `@coderabbitai read src/utils.ts and generate unit testing code.`
  - `@coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.`
  - `@coderabbitai help me debug CodeRabbit configuration file.`

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

### CodeRabbit Commands (Invoked using PR comments)

- `@coderabbitai pause` to pause the reviews on a PR.
- `@coderabbitai resume` to resume the paused reviews.
- `@coderabbitai review` to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
- `@coderabbitai full review` to do a full review from scratch and review all the files again.
- `@coderabbitai summary` to regenerate the summary of the PR.
- `@coderabbitai generate docstrings` to [generate docstrings](https://docs.coderabbit.ai/finishing-touches/docstrings) for this PR.
- `@coderabbitai resolve` resolve all the CodeRabbit review comments.
- `@coderabbitai configuration` to show the current CodeRabbit configuration for the repository.
- `@coderabbitai help` to get help.

### Other keywords and placeholders

- Add `@coderabbitai ignore` anywhere in the PR description to prevent this PR from being reviewed.
- Add `@coderabbitai summary` to generate the high-level summary at a specific location in the PR description.
- Add `@coderabbitai` anywhere in the PR title to generate the title automatically.

### CodeRabbit Configuration File (`.coderabbit.yaml`)

- You can programmatically configure CodeRabbit by adding a `.coderabbit.yaml` file to the root of your repository.
- Please see the [configuration documentation](https://docs.coderabbit.ai/guides/configure-coderabbit) for more information.
- If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: `# yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json`

### Documentation and Community

- Visit our [Documentation](https://docs.coderabbit.ai) for detailed information on how to use CodeRabbit.
- Join our [Discord Community](http://discord.gg/coderabbit) to get help, request features, and share feedback.
- Follow us on [X/Twitter](https://twitter.com/coderabbitai) for updates and announcements.

</details>

<!-- tips_end -->
author_association: CONTRIBUTOR
user: login: github-actions[bot]
created_at: 2025-03-05T08:36:30Z
body: #### Tofu Plan Output - observability_stack_ctrl_plane_infra
```

[0m[1m[32mNo changes.[0m[1m Your infrastructure matches the configuration.[0m

[0mOpenTofu has compared your real infrastructure against your configuration and
found no differences, so no changes are needed.
```
    
author_association: NONE
user: login: github-actions[bot]
created_at: 2025-03-05T08:36:35Z
body: #### Tofu Plan Output - observability_stack_ctrl_plane_k8s
```

[0m[1m[32mNo changes.[0m[1m Your infrastructure matches the configuration.[0m

[0mOpenTofu has compared your real infrastructure against your configuration and
found no differences, so no changes are needed.
```
    
author_association: NONE