Repository: keephq/keep
Pull request: feat: added support for step retry
<!-- 
Thanks for creating this pull request 🤗

Please make sure that the pull request is limited to one type (docs, feature, etc.) and keep it as small as possible. You can open multiple prs instead of opening a huge one.
-->

<!-- If this pull request closes an issue, please mention the issue number below -->
Closes #239 

## 📑 Description
Added support for retrying step on failure for specified count and interval.
Example:
```
# Alert if a result queried from the DB is above a certain thershold.
alert:
  id: raw-sql-query
  description: Monitor that time difference is no more than 1 hour
  steps:
    - name: get-max-datetime
      provider:
        type: mysql
        config: "{{ providers.mysql-prod }}"
        with:
          # Get max(datetime) from the random table
          query: "SELECT MAX(datetime) FROM demo_table LIMIT 1"
        on-failure:
          retry: 
            count: 4
            interval: 15
  actions:
    - name: trigger-ntfy
      condition:
      - name: threshold-condition
        type: threshold
```

<!-- You can also choose to add a list of changes and if they have been completed or not by using the markdown to-do list syntax
- [ ] Not Completed
- [x] Completed
-->

## ✅ Checks
<!-- Make sure your pr passes the CI checks and do check the following fields as needed - -->
- [x] My pull request adheres to the code style of this project
- [x] My code requires changes to the documentation
- [x] I have updated the documentation as required
- [ ] All the tests have passed

/claim #239

Problem statement:
Step retry mechanism
**Is your feature request related to a problem? Please describe.**
Sometimes when a step fails to run (e.g. a simple HTTP request), I'd like to be able to have a retry mechanism before failing the step.

**Describe the solution you'd like**
An optional retry mechanism for steps to define what to do when a step fails.

**Describe alternatives you've considered**
-

**Additional context**
Requested by @configurator

Hints from issue discussion:
@talboren correct if I misunderstoof your concern, basically you want an option like a `Retry button` so that you can restart the failed step correct? 

can you provide a screenshot or design?
@Abhishek-90 actually I meant the YAML spec.

This is something I've imagined that could potentially work:
```yaml
# Alert if a result queried from the DB is above a certain thershold.
alert:
  id: raw-sql-query
  description: Monitor that time difference is no more than 1 hour
  steps:
    - name: get-max-datetime
      provider:
        type: mysql
        config: "{{ providers.mysql-prod }}"
        with:
          # Get max(datetime) from the random table
          query: "SELECT MAX(datetime) FROM demo_table LIMIT 1"
        on-failure:
            retry: 3
  actions:
    - name: trigger-ntfy
      condition:
      - name: threshold-condition
        type: threshold
...
```

If step `get-max-datetime` fails, it'll retry the step 3 times before failing the workflow.
@talboren Is there any architecture design for how the workflow is processed? Like parsing -> storing workflow details -> executing steps/actions ?
@devblin not yet actually. We’re working on adding high level architecture design to our docs, probably next week 🤞🏻
Until then, I’d be more than happy to hop on a quick call and do a guided tour ;)

Saw you just joined our Slack! Let’s talk over a DM
/bounty 20
@devblin please make sure you claim this bounty if you're working on it :)
@talboren can you please assign the issue to me?
/attempt #239

<details id="algora-options">
  <summary>Options</summary>
  <ul>
    <li>
      <a href="https://console.algora.io/api/bounties/clmdc0ys10002me0fbuf89e0v/cancel-attempt">
        Cancel my attempt
      </a>
    </li>
  </ul>
</details>

Commit message: Merge branch 'main' into task/239

Patch to review:
diff --git a/keep/step/step.py b/keep/step/step.py
index daec69d81d..2e9fd56065 100644
--- a/keep/step/step.py
+++ b/keep/step/step.py
@@ -1,6 +1,7 @@
 import asyncio
 import inspect
 import logging
+import time
 from dataclasses import field
 from enum import Enum
 
@@ -35,11 +36,15 @@ def __init__(
         self.step_type = step_type
         self.provider = provider
         self.provider_parameters = provider_parameters
+        self.on_failure = self.config.get("on-failure", {})
         self.context_manager = context_manager
         self.io_handler = IOHandler(context_manager)
         self.conditions = self.config.get("condition", [])
         self.conditions_results = {}
         self.logger = context_manager.get_logger()
+        self.__retry = self.on_failure.get("retry", {})
+        self.__retry_count = self.__retry.get("count", 0)
+        self.__retry_interval = self.__retry.get("interval", 0)
 
     @property
     def foreach(self):
@@ -167,28 +172,34 @@ def _run_single(self):
             result = self._run_single_async()
         # else, just run the provider
         else:
-            try:
-                rendered_providers_parameters = {}
-                for parameter in self.provider_parameters:
-                    rendered_providers_parameters[parameter] = self.io_handler.render(
-                        self.provider_parameters[parameter]
+            for curr_retry_count in range(self.__retry_count + 1):
+                try:
+                    rendered_providers_parameters = {}
+                    for parameter in self.provider_parameters:
+                        rendered_providers_parameters[parameter] = self.io_handler.render(
+                            self.provider_parameters[parameter]
+                        )
+
+                    if self.step_type == StepType.STEP:
+                        step_output = self.provider.query(**rendered_value)
+                        self.context_manager.set_step_context(
+                            self.step_id, results=step_output, foreach=self.foreach
+                        )
+                    else:
+                        self.provider.notify(**rendered_value)
+
+                    extra_context = self.provider.expose()
+                    rendered_providers_parameters.update(extra_context)
+                    self.context_manager.set_step_provider_paremeters(
+                        self.step_id, rendered_providers_parameters
                     )
-
-                if self.step_type == StepType.STEP:
-                    step_output = self.provider.query(**rendered_value)
-                    self.context_manager.set_step_context(
-                        self.step_id, results=step_output, foreach=self.foreach
-                    )
-                else:
-                    self.provider.notify(**rendered_value)
-
-                extra_context = self.provider.expose()
-                rendered_providers_parameters.update(extra_context)
-                self.context_manager.set_step_provider_paremeters(
-                    self.step_id, rendered_providers_parameters
-                )
-            except Exception as e:
-                raise StepError(e)
+                except Exception as e:
+                    if curr_retry_count == self.__retry_count:
+                        raise StepError(e)
+                    else:
+                        self.logger.info(
+                            "Retrying running %s step after %s second(s)...", self.step_id, self.__retry_interval)
+                        time.sleep(self.__retry_interval)
 
             return True
 


Code review comments:
diff_hunk: @@ -167,28 +172,34 @@ def _run_single(self):
             result = self._run_single_async()
         # else, just run the provider
         else:
-            try:
-                rendered_providers_parameters = {}
-                for parameter in self.provider_parameters:
-                    rendered_providers_parameters[parameter] = self.io_handler.render(
-                        self.provider_parameters[parameter]
+            for curr_retry_count in range(self.__retry_count + 1):
+                try:
+                    rendered_providers_parameters = {}
+                    for parameter in self.provider_parameters:
+                        rendered_providers_parameters[parameter] = self.io_handler.render(
+                            self.provider_parameters[parameter]
+                        )
+
+                    if self.step_type == StepType.STEP:
+                        step_output = self.provider.query(**rendered_value)
+                        self.context_manager.set_step_context(
+                            self.step_id, results=step_output, foreach=self.foreach
+                        )
+                    else:
+                        self.provider.notify(**rendered_value)
+
+                    extra_context = self.provider.expose()
+                    rendered_providers_parameters.update(extra_context)
+                    self.context_manager.set_step_provider_paremeters(
+                        self.step_id, rendered_providers_parameters
                     )
-
-                if self.step_type == StepType.STEP:
-                    step_output = self.provider.query(**rendered_value)
-                    self.context_manager.set_step_context(
-                        self.step_id, results=step_output, foreach=self.foreach
-                    )
-                else:
-                    self.provider.notify(**rendered_value)
-
-                extra_context = self.provider.expose()
-                rendered_providers_parameters.update(extra_context)
-                self.context_manager.set_step_provider_paremeters(
-                    self.step_id, rendered_providers_parameters
-                )
-            except Exception as e:
-                raise StepError(e)
+                except Exception as e:
+                    if curr_retry_count == self.__retry_count:
+                        raise StepError(e)
+                    else:
+                        self.logger.info(
path: keep/step/step.py
text: @user1:
anyway it worth adding the original exception so we will have some trace (str(e))