Repository: keephq/keep
Pull request: feat: Add deduplication rules provisioning mechanism from KEEP_DEDUPLICATION_RULES env variable
<!-- 
Thanks for creating this pull request 🤗

Please make sure that the pull request is limited to one type (docs, feature, etc.) and keep it as small as possible. You can open multiple prs instead of opening a huge one.
-->

<!-- If this pull request closes an issue, please mention the issue number below -->
Closes #2872 

## 📑 Description
<!-- Add a brief description of the pr -->
Enable provisioning of deduplication rules during deployment through KEEP_DEDUPLICATION_RULES env variable 
<!-- You can also choose to add a list of changes and if they have been completed or not by using the markdown to-do list syntax
- [ ] Not Completed
- [x] Completed
-->

## ✅ Checks
<!-- Make sure your pr passes the CI checks and do check the following fields as needed - -->
- [x] My pull request adheres to the code style of this project
- [x] My code requires changes to the documentation
- [x] I have updated the documentation as required
- [x] All the tests have passed

## ℹ Additional Information
<!-- Any additional information like breaking changes, dependencies added, screenshots, comparisons between new and old behavior, etc. -->


Problem statement:
[➕ Feature]: Create Deduplication rule on provision
**Is your feature request related to a problem? Please describe.**
I'm trying to set up Keep using as much IaC as possible, and it's only possible to create dedupe rules in the console or the API. It'd be great for my use case if they could be set up via configuration.

**Describe the solution you'd like**
Ideally, they should be able to be provisioned resources, in the same manner as workflows, providers, and dashboards.

**Describe alternatives you've considered**
It'd also be cool if Keep resources and rules could be deployed with Terraform

**Additional context**
Add any other context or screenshots about the feature request here.

Hints from issue discussion:
@fues-joe1 We have a terraform module contributed by the amazing @pehlicd, but it's not too maintained, we need to pick it up: https://github.com/pehlicd/terraform-provider-keep
I'll take care of provisioning for deduplication rules for now
Hi!
I have implemented this feature and added documentation. Details in PR - https://github.com/keephq/keep/pull/2877

Commit message: adjust configuration

Patch to review:
diff --git a/docs/deployment/provision/overview.mdx b/docs/deployment/provision/overview.mdx
index 7d6226f6ab..290030d952 100644
--- a/docs/deployment/provision/overview.mdx
+++ b/docs/deployment/provision/overview.mdx
@@ -8,10 +8,11 @@ Keep supports various deployment and provisioning strategies to accommodate diff
 
 Keep offers three main provisioning options:
 
-1. [**Provider Provisioning**](/deployment/provision/provider) - Set up and manage data providers for Keep.
+1. [**Provider Provisioning**](/deployment/provision/provider) - Set up and manage data providers with their deduplication rules for Keep.
 2. [**Workflow Provisioning**](/deployment/provision/workflow) - Configure and manage workflows within Keep.
 3. [**Dashboard Provisioning**](/deployment/provision/dashboard) - Configure and manage dashboards within Keep.
 
+
 Choosing the right provisioning strategy depends on your specific use case, deployment environment, and scalability requirements. You can read more about each provisioning option in their respective sections.
 
 ### How To Configure Provisioning
@@ -23,10 +24,11 @@ Choosing the right provisioning strategy depends on your specific use case, depl
 
 Provisioning in Keep is controlled through environment variables and configuration files. The main environment variables for provisioning are:
 
-| Provisioning Type | Environment Variable       | Purpose                                                |
-| ----------------- | -------------------------- | ------------------------------------------------------ |
-| **Provider**      | `KEEP_PROVIDERS`           | JSON string containing provider configurations         |
-| **Workflow**      | `KEEP_WORKFLOWS_DIRECTORY` | Directory path containing workflow configuration files |
-| **Dashboard**     | `KEEP_DASHBOARDS`          | JSON string containing dashboard configurations        |
+| Provisioning Type      | Environment Variable           | Purpose                                                                 |
+| ---------------------- | ------------------------------ | ----------------------------------------------------------------------- |
+| **Provider**           | `KEEP_PROVIDERS`               | JSON string containing provider configurations with deduplication rules |
+| **Workflow**           | `KEEP_WORKFLOWS_DIRECTORY`     | Directory path containing workflow configuration files                  |
+| **Dashboard**          | `KEEP_DASHBOARDS`              | JSON string containing dashboard configurations                         |
+
 
 For more details on each provisioning strategy, including setup instructions and implications, refer to the respective sections.
diff --git a/docs/deployment/provision/provider.mdx b/docs/deployment/provision/provider.mdx
index f6993aabf3..1206a76fce 100644
--- a/docs/deployment/provision/provider.mdx
+++ b/docs/deployment/provision/provider.mdx
@@ -8,15 +8,31 @@ Provider provisioning in Keep allows you to set up and manage data providers dyn
 
 ### Configuring Providers
 
-To provision providers, set the `KEEP_PROVIDERS` environment variable with a JSON string containing the provider configurations. Here's an example:
+To provision providers and deduplication rules for them, set the `KEEP_PROVIDERS` environment variable. This can be done in two ways:
+1. Directly with a JSON string containing the providers configurations.
+2. With the path to a JSON file that contains the providers configurations.
 
+Providers provisioning JSON example:
 ```json
 {
   "keepVictoriaMetrics": {
     "type": "victoriametrics",
     "authentication": {
       "VMAlertHost": "http://localhost",
-      "VMAlertPort": 1234
+      "VMAlertPort": 1234,
+      "deduplication_rules": {
+        "deduplication rule name example 1": {
+          "description": "deduplication rule name example 1",
+          "fingerprint_fields": ["fingerprint", "source", "service"],
+          "full_deduplication": true,
+          "ignore_fields": ["name", "lastReceived"]
+        },
+        "deduplication rule name example 2": {
+          "description": "deduplication rule name example 2",
+          "fingerprint_fields": ["fingerprint", "source", "service"],
+          "full_deduplication": false,
+        }
+      }
     }
   },
   "keepClickhouse1": {
diff --git a/docs/mint.json b/docs/mint.json
index 59101220e7..b941d3adc8 100644
--- a/docs/mint.json
+++ b/docs/mint.json
@@ -65,7 +65,6 @@
       "group": "AIOps",
       "pages": [
         "overview/correlation",
-        "overview/deduplication",
         "overview/enrichment/extraction",
         "overview/enrichment/mapping",
         "overview/servicetopology",
@@ -231,7 +230,8 @@
             "deployment/provision/overview",
             "deployment/provision/provider",
             "deployment/provision/workflow",
-            "deployment/provision/dashboard"
+            "deployment/provision/dashboard",
+            "deployment/provision/deduplication-rule"
           ]
         },
         "deployment/secret-store",
diff --git a/keep-ui/app/(keep)/deduplication/DeduplicationSidebar.tsx b/keep-ui/app/(keep)/deduplication/DeduplicationSidebar.tsx
index bdde5f0187..b1950094a4 100644
--- a/keep-ui/app/(keep)/deduplication/DeduplicationSidebar.tsx
+++ b/keep-ui/app/(keep)/deduplication/DeduplicationSidebar.tsx
@@ -241,6 +241,22 @@ const DeduplicationSidebar: React.FC<DeduplicationSidebarProps> = ({
         </div>
       )}
 
+      {selectedDeduplicationRule?.is_provisioned && (
+        <div className="flex flex-col">
+          <Callout
+            className="mb-4 py-8"
+            title="Editing a Provisioned Rule"
+            icon={ExclamationTriangleIcon}
+            color="orange"
+          >
+            <Text>
+              Editing a provisioned deduplication rule is not allowed.
+              Please contact your system administrator for more information.
+            </Text>
+          </Callout>
+        </div>
+      )}
+
       <form
         onSubmit={handleSubmit(onFormSubmit)}
         className="mt-4 flex flex-col h-full"
@@ -256,6 +272,7 @@ const DeduplicationSidebar: React.FC<DeduplicationSidebarProps> = ({
                   name="name"
                   control={control}
                   rules={{ required: "Rule name is required" }}
+                  disabled={!!selectedDeduplicationRule?.is_provisioned}
                   render={({ field }) => (
                     <TextInput
                       {...field}
@@ -273,6 +290,7 @@ const DeduplicationSidebar: React.FC<DeduplicationSidebarProps> = ({
                   name="description"
                   control={control}
                   rules={{ required: "Description is required" }}
+                  disabled={!!selectedDeduplicationRule?.is_provisioned}
                   render={({ field }) => (
                     <TextInput
                       {...field}
@@ -306,7 +324,7 @@ const DeduplicationSidebar: React.FC<DeduplicationSidebarProps> = ({
                   render={({ field }) => (
                     <Select<ProviderOption, false, GroupBase<ProviderOption>>
                       {...field}
-                      isDisabled={!!selectedDeduplicationRule?.default}
+                      isDisabled={!!selectedDeduplicationRule?.default || selectedDeduplicationRule?.is_provisioned}
                       options={alertProviders.map((provider) => ({
                         value: `${provider.type}_${provider.id}`,
                         label: provider.details?.name || provider.id || "main",
@@ -380,6 +398,7 @@ const DeduplicationSidebar: React.FC<DeduplicationSidebarProps> = ({
                   render={({ field }) => (
                     <MultiSelect
                       {...field}
+                      isDisabled={!!selectedDeduplicationRule?.is_provisioned}
                       options={availableFields.map((fieldName) => ({
                         value: fieldName,
                         label: fieldName,
@@ -416,7 +435,11 @@ const DeduplicationSidebar: React.FC<DeduplicationSidebarProps> = ({
                     name="full_deduplication"
                     control={control}
                     render={({ field }) => (
-                      <Switch checked={field.value} onChange={field.onChange} />
+                      <Switch 
+                        disabled={!!selectedDeduplicationRule?.is_provisioned} 
+                        checked={field.value}
+                        onChange={field.onChange} 
+                      />
                     )}
                   />
                   <span className="text-sm font-medium text-gray-700 flex items-center">
@@ -454,6 +477,7 @@ const DeduplicationSidebar: React.FC<DeduplicationSidebarProps> = ({
                     render={({ field }) => (
                       <MultiSelect
                         {...field}
+                        isDisabled={!!selectedDeduplicationRule?.is_provisioned}
                         options={availableFields.map((fieldName) => ({
                           value: fieldName,
                           label: fieldName,
@@ -502,7 +526,11 @@ const DeduplicationSidebar: React.FC<DeduplicationSidebarProps> = ({
           >
             Cancel
           </Button>
-          <Button color="orange" type="submit" disabled={isSubmitting}>
+          <Button
+            color="orange"
+            type="submit"
+            disabled={isSubmitting || selectedDeduplicationRule?.is_provisioned}
+          >
             {isSubmitting ? "Saving..." : "Save Rule"}
           </Button>
         </div>
diff --git a/keep-ui/app/(keep)/deduplication/DeduplicationTable.tsx b/keep-ui/app/(keep)/deduplication/DeduplicationTable.tsx
index 34ab80e236..909ea6ee41 100644
--- a/keep-ui/app/(keep)/deduplication/DeduplicationTable.tsx
+++ b/keep-ui/app/(keep)/deduplication/DeduplicationTable.tsx
@@ -120,6 +120,18 @@ export const DeduplicationTable: React.FC<DeduplicationTableProps> = ({
       "Represents the percentage of alerts successfully deduplicated. Higher values indicate better deduplication efficiency, meaning fewer redundant alerts.",
   };
 
+  function resolveDeleteButtonTooltip(deduplicationRule: DeduplicationRule): string {
+    if (deduplicationRule.default) {
+      return "Cannot delete default rule";
+    }
+
+    if (deduplicationRule.is_provisioned) {
+      return "Cannot delete provisioned rule.";
+    }
+
+    return "Delete Rule"
+  }
+
   const DEDUPLICATION_TABLE_COLS = useMemo(
     () => [
       columnHelper.accessor("provider_type", {
@@ -265,11 +277,9 @@ export const DeduplicationTable: React.FC<DeduplicationTableProps> = ({
               variant="secondary"
               icon={TrashIcon}
               tooltip={
-                info.row.original.default
-                  ? "Cannot delete default rule"
-                  : "Delete Rule"
+                resolveDeleteButtonTooltip(info.row.original)
               }
-              disabled={info.row.original.default}
+              disabled={info.row.original.default || info.row.original.is_provisioned}
               onClick={(e) => handleDeleteRule(info.row.original, e)}
             />
           </div>
diff --git a/keep-ui/app/(keep)/deduplication/models.tsx b/keep-ui/app/(keep)/deduplication/models.tsx
index 8e5e0d3ec4..a201cd5137 100644
--- a/keep-ui/app/(keep)/deduplication/models.tsx
+++ b/keep-ui/app/(keep)/deduplication/models.tsx
@@ -17,4 +17,5 @@ export interface DeduplicationRule {
   // full_deduplication is true if the deduplication rule is a full deduplication rule
   full_deduplication: boolean;
   ignore_fields: string[];
+  is_provisioned: boolean;
 }
diff --git a/keep/api/alert_deduplicator/alert_deduplicator.py b/keep/api/alert_deduplicator/alert_deduplicator.py
index 0ed44c4af9..09e30ab9fe 100644
--- a/keep/api/alert_deduplicator/alert_deduplicator.py
+++ b/keep/api/alert_deduplicator/alert_deduplicator.py
@@ -15,6 +15,7 @@
     get_all_deduplication_rules,
     get_all_deduplication_stats,
     get_custom_deduplication_rule,
+    get_deduplication_rule_by_id,
     get_last_alert_hash_by_fingerprint,
     update_deduplication_rule,
 )
@@ -263,6 +264,7 @@ def _get_default_full_deduplication_rule(
             ingested=0,
             dedup_ratio=0.0,
             enabled=True,
+            is_provisioned=False
         )
 
     def get_deduplications(self) -> list[DeduplicationRuleDto]:
@@ -310,6 +312,7 @@ def get_deduplications(self) -> list[DeduplicationRuleDto]:
                 ingested=0,
                 dedup_ratio=0.0,
                 enabled=rule.enabled,
+                is_provisioned=rule.is_provisioned,
             )
             for rule in custom_deduplications
         ]
@@ -478,6 +481,19 @@ def create_deduplication_rule(
     def update_deduplication_rule(
         self, rule_id: str, rule: DeduplicationRuleRequestDto, updated_by: str
     ) -> DeduplicationRuleDto:
+        """
+        Updates an existing deduplication rule or creates a new one if the rule is a default rule.
+        Args:
+            rule_id (str): The ID of the deduplication rule to update.
+            rule (DeduplicationRuleRequestDto): The new deduplication rule data.
+            updated_by (str): The identifier of the user who is updating the rule.
+        Returns:
+            DeduplicationRuleDto: The updated deduplication rule.
+        Raises:
+            HTTPException 404: If the deduplication rule is not found (404)
+            HTTPException 409: if a provisioned rule is attempted to be updated (409).
+        """
+
         # check if this is a default rule
         default_rule_id = self._generate_uuid(rule.provider_id, rule.provider_type)
         # if its a default, we need to override and create a new rule
@@ -486,6 +502,20 @@ def update_deduplication_rule(
             rule_dto = self.create_deduplication_rule(rule, updated_by)
             self.logger.info("Default rule updated")
             return rule_dto
+        
+        rule_before_update = get_deduplication_rule_by_id(self.tenant_id, rule_id)
+        
+        if not rule_before_update:
+            raise HTTPException(
+                status_code=404,
+                detail="Deduplication rule not found",
+            )
+        
+        if rule_before_update.is_provisioned:
+            raise HTTPException(
+                status_code=409,
+                detail="Provisioned deduplication rule cannot be updated",
+            )
 
         # else, use the db function to update an existing deduplication rule
         updated_rule = update_deduplication_rule(
@@ -506,7 +536,34 @@ def update_deduplication_rule(
         return updated_rule
 
     def delete_deduplication_rule(self, rule_id: str) -> bool:
+        """
+        Deletes a deduplication rule by its ID.
+        Args:
+            rule_id (str): The ID of the deduplication rule to be deleted.
+        Returns:
+            bool: True if the deduplication rule was successfully deleted, False otherwise.
+        Raises:
+            HTTPException 404: If the deduplication rule is not found.
+            HTTPException 409: If the deduplication rule is provisioned and cannot be deleted.
+        """
+
         # Use the db function to delete a deduplication rule
+        deduplication_rule_to_be_deleted = get_deduplication_rule_by_id(
+            self.tenant_id, rule_id
+        )
+
+        if not deduplication_rule_to_be_deleted:
+            raise HTTPException(
+                status_code=404,
+                detail="Deduplication rule not found",
+            )
+        
+        if deduplication_rule_to_be_deleted.is_provisioned:
+            raise HTTPException(
+                status_code=409,
+                detail="Provisioned deduplication rule cannot be deleted",
+            )
+
         success = delete_deduplication_rule(rule_id=rule_id, tenant_id=self.tenant_id)
 
         return success
diff --git a/keep/api/alert_deduplicator/deduplication_rules_provisioning.py b/keep/api/alert_deduplicator/deduplication_rules_provisioning.py
new file mode 100644
index 0000000000..64df28cb11
--- /dev/null
+++ b/keep/api/alert_deduplicator/deduplication_rules_provisioning.py
@@ -0,0 +1,178 @@
+import json
+import logging
+import re
+
+
+from keep.api.core.config import config
+import keep.api.core.db as db
+from keep.providers.providers_factory import ProvidersFactory
+
+logger = logging.getLogger(__name__)
+
+def provision_deduplication_rules_from_env(tenant_id: str):
+    """
+    Provisions deduplication rules from environment variables for a given tenant.
+    This function reads deduplication rules from environment variables, validates them,
+    and then provisions them into the database. It handles the following:
+    - Deletes deduplication rules from the database that are not present in the environment variables.
+    - Updates existing deduplication rules in the database if they are present in the environment variables.
+    - Creates new deduplication rules in the database if they are not already present.
+    Args:
+        tenant_id (str): The ID of the tenant for which deduplication rules are being provisioned.
+    Raises:
+        ValueError: If the deduplication rules from the environment variables are invalid.
+    """
+
+    deduplication_rules_from_env_dict = get_deduplication_rules_to_provision()
+
+    if not deduplication_rules_from_env_dict:
+        logger.info("No deduplication rules found in env. Nothing to provision.")
+        return
+    
+    enrich_with_providers_info(deduplication_rules_from_env_dict, tenant_id)
+
+    all_deduplication_rules_from_db = db.get_all_deduplication_rules(tenant_id)
+    provisioned_deduplication_rules = [
+        rule for rule in all_deduplication_rules_from_db if rule.is_provisioned
+    ]
+    provisioned_deduplication_rules_from_db_dict = {
+        rule.name: rule for rule in provisioned_deduplication_rules
+    }
+    actor = "system"
+
+    # delete rules that are not in the env
+    for provisioned_deduplication_rule in provisioned_deduplication_rules:
+        if (
+            str(provisioned_deduplication_rule.name)
+            not in deduplication_rules_from_env_dict
+        ):
+            logger.info(
+                "Deduplication rule with name '%s' is not in the env, deleting from DB",
+                provisioned_deduplication_rule.name,
+            )
+            db.delete_deduplication_rule(rule_id=str(provisioned_deduplication_rule.id), tenant_id=tenant_id)
+
+    for deduplication_rule_to_provision in deduplication_rules_from_env_dict.values():
+        if (
+            deduplication_rule_to_provision.get("name")
+            in provisioned_deduplication_rules_from_db_dict
+        ):
+
+            logger.info(
+                "Deduplication rule with name '%s' already exists, updating in DB",
+                deduplication_rule_to_provision.get("name"),
+            )
+            db.update_deduplication_rule(
+                tenant_id=tenant_id,
+                rule_id=str(
+                    provisioned_deduplication_rules_from_db_dict.get(
+                        deduplication_rule_to_provision.get("name")
+                    ).id
+                ),
+                name=deduplication_rule_to_provision.get("name"),
+                description=deduplication_rule_to_provision.get("description"),
+                provider_id=deduplication_rule_to_provision.get("provider_id"),
+                provider_type=deduplication_rule_to_provision.get("provider_type"),
+                last_updated_by=actor,
+                enabled=True,
+                fingerprint_fields=deduplication_rule_to_provision.get("fingerprint_fields"),
+                full_deduplication=deduplication_rule_to_provision.get("full_deduplication"),
+                ignore_fields=deduplication_rule_to_provision.get("ignore_fields") or [],
+                priority=0,
+            )
+            continue
+
+        logger.info(
+            "Deduplication rule with name '%s' does not exist, creating in DB",
+            deduplication_rule_to_provision.get("name"),
+        )
+        db.create_deduplication_rule(
+            tenant_id=tenant_id,
+            name=deduplication_rule_to_provision.get("name"),
+            description=deduplication_rule_to_provision.get("description"),
+            provider_id=deduplication_rule_to_provision.get("provider_id"),
+            provider_type=deduplication_rule_to_provision.get("provider_type"),
+            created_by=actor,
+            enabled=True,
+            fingerprint_fields=deduplication_rule_to_provision.get("fingerprint_fields"),
+            full_deduplication=deduplication_rule_to_provision.get("full_deduplication"),
+            ignore_fields=deduplication_rule_to_provision.get("ignore_fields") or [],
+            priority=0,
+            is_provisioned=True,
+        )
+
+
+def enrich_with_providers_info(deduplication_rules: list[dict[str,any]], tenant_id: str):
+    """
+    Enriches passed deduplication rules with provider ID and type information.
+    Args:
+        deduplication_rules (list[dict]): A list of deduplication rules to be enriched.
+        tenant_id (str): The ID of the tenant for which deduplication rules are being provisioned.
+    """
+
+    installed_providers = ProvidersFactory.get_installed_providers(tenant_id)
+    linked_providers = ProvidersFactory.get_linked_providers(tenant_id)
+    all_providers = installed_providers + linked_providers
+    all_providers_dict = {f"{provider.type}_{provider.display_name}": provider for provider in all_providers}
+
+    for rule_name, rule in deduplication_rules.items():
+        print('f')
+        provider_dict_key = f"{rule.get('provider_type')}_{rule.get('provider_name')}"
+        provider = all_providers_dict.get(provider_dict_key)
+        rule["provider_id"] = provider.id
+        rule["provider_type"] = provider.type
+
+
+def get_deduplication_rules_to_provision() -> dict[str, dict]:
+    """
+    Reads deduplication rules from an environment variable and returns them as a dictionary.
+    The function checks if the environment variable `KEEP_DEDUPLICATION_RULES` contains a path to a JSON file
+    or a JSON string. If it is a path, it reads the file and parses the JSON content. If it is a JSON string,
+    it parses the string directly.
+    Returns:
+        dict[str, DeduplicationRuleRequestDto]: A dictionary where the keys are rule names and the values are
+        DeduplicationRuleRequestDto objects.
+    Raises:
+        Exception: If there is an error parsing the JSON content from the file or the environment variable.
+    """
+
+    env_var_key = "KEEP_PROVIDERS"
+    deduplication_rules_from_env_var = config(key=env_var_key, default=None)
+
+    if not deduplication_rules_from_env_var:
+        return None
+
+    # check if env var is absolute or relative path to a deduplication rules json file
+    if re.compile(r"^(\/|\.\/|\.\.\/).*\.json$").match(
+        deduplication_rules_from_env_var
+    ):
+        with open(
+            file=deduplication_rules_from_env_var, mode="r", encoding="utf8"
+        ) as file:
+            try:
+                deduplication_rules_from_env_json: dict = json.loads(file.read())
+            except json.JSONDecodeError as e:
+                raise Exception(
+                    f"Error parsing deduplication rules from file {deduplication_rules_from_env_var}: {e}"
+                ) from e
+    else:
+        try:
+            deduplication_rules_from_env_json = json.loads(deduplication_rules_from_env_var)
+        except json.JSONDecodeError as e:
+            raise Exception(
+                f"Error parsing deduplication rules from env var {env_var_key}: {e}"
+            ) from e
+        
+    deduplication_rules_dict: dict[str, dict] = {}
+
+    for provider_name, provider_config in deduplication_rules_from_env_json.items():
+        for rule_name, rule_config in provider_config.get("deduplication_rules", {}).items():
+            rule_config["name"] = rule_name
+            rule_config["provider_name"] = provider_name
+            rule_config["provider_type"] = provider_config.get("type")
+            deduplication_rules_dict[rule_name] = rule_config
+
+    if not deduplication_rules_dict:
+        return None
+
+    return deduplication_rules_dict
diff --git a/keep/api/config.py b/keep/api/config.py
index e567bb9306..f4121d0aec 100644
--- a/keep/api/config.py
+++ b/keep/api/config.py
@@ -1,6 +1,7 @@
 import logging
 import os
 
+from keep.api.alert_deduplicator.deduplication_rules_provisioning import provision_deduplication_rules_from_env
 import keep.api.logging
 from keep.api.api import AUTH_TYPE
 from keep.api.core.db_on_start import migrate_db, try_create_single_tenant
@@ -30,6 +31,9 @@ def provision_resources():
         logger.info("Workflows provisioned successfully")
         provision_dashboards(SINGLE_TENANT_UUID)
         logger.info("Dashboards provisioned successfully")
+        logger.info("Provisioning deduplication rules")
+        provision_deduplication_rules_from_env(SINGLE_TENANT_UUID)
+        logger.info("Deduplication rules provisioned successfully")
     else:
         logger.info("Provisioning resources is disabled")
 
diff --git a/keep/api/core/db.py b/keep/api/core/db.py
index 0cd8ebe8e4..93a0f08cc4 100644
--- a/keep/api/core/db.py
+++ b/keep/api/core/db.py
@@ -1903,6 +1903,19 @@ def get_all_deduplication_rules(tenant_id):
         ).all()
     return rules
 
+def get_deduplication_rule_by_id(tenant_id, rule_id: str):
+    rule_uuid = __convert_to_uuid(rule_id)
+    if not rule_uuid:
+        return None
+
+    with Session(engine) as session:
+        rules = session.exec(
+            select(AlertDeduplicationRule)
+            .where(AlertDeduplicationRule.tenant_id == tenant_id)
+            .where(AlertDeduplicationRule.id == rule_uuid)
+        ).first()
+    return rules
+
 
 def get_custom_deduplication_rule(tenant_id, provider_id, provider_type):
     with Session(engine) as session:
@@ -1927,6 +1940,7 @@ def create_deduplication_rule(
     full_deduplication: bool = False,
     ignore_fields: list[str] = [],
     priority: int = 0,
+    is_provisioned: bool = False
 ):
     with Session(engine) as session:
         new_rule = AlertDeduplicationRule(
@@ -1942,6 +1956,7 @@ def create_deduplication_rule(
             full_deduplication=full_deduplication,
             ignore_fields=ignore_fields,
             priority=priority,
+            is_provisioned=is_provisioned,
         )
         session.add(new_rule)
         session.commit()
diff --git a/keep/api/models/alert.py b/keep/api/models/alert.py
index fd5b4c0b06..e4d2dac26d 100644
--- a/keep/api/models/alert.py
+++ b/keep/api/models/alert.py
@@ -599,6 +599,7 @@ class DeduplicationRuleDto(BaseModel):
     fingerprint_fields: list[str]
     full_deduplication: bool
     ignore_fields: list[str]
+    is_provisioned: bool
 
 
 class DeduplicationRuleRequestDto(BaseModel):
diff --git a/keep/api/models/db/alert.py b/keep/api/models/db/alert.py
index 1e2cb8d5c8..31d83fe47f 100644
--- a/keep/api/models/db/alert.py
+++ b/keep/api/models/db/alert.py
@@ -292,6 +292,7 @@ class AlertDeduplicationRule(SQLModel, table=True):
     full_deduplication: bool = Field(default=False)
     ignore_fields: list[str] = Field(sa_column=Column(JSON), default=[])
     priority: int = Field(default=0)  # for future use
+    is_provisioned: bool = Field(default=False)
 
     class Config:
         arbitrary_types_allowed = True
diff --git a/keep/api/models/db/migrations/versions/2024-12-23-18-49_4f8c4b185d5b.py b/keep/api/models/db/migrations/versions/2024-12-23-18-49_4f8c4b185d5b.py
new file mode 100644
index 0000000000..c0e03a0e35
--- /dev/null
+++ b/keep/api/models/db/migrations/versions/2024-12-23-18-49_4f8c4b185d5b.py
@@ -0,0 +1,42 @@
+"""Add is_provisioned column for DeduplicationRule table
+
+Revision ID: 4f8c4b185d5b
+Revises: 0c5e002094a9
+Create Date: 2024-12-23 18:49:00.882402
+
+"""
+
+import sqlalchemy as sa
+from alembic import op
+
+# revision identifiers, used by Alembic.
+revision = "4f8c4b185d5b"
+down_revision = "0c5e002094a9"
+branch_labels = None
+depends_on = None
+
+
+def upgrade() -> None:
+    # ### commands auto generated by Alembic - please adjust! ###
+    # Add the new column with a server default
+    with op.batch_alter_table("alertdeduplicationrule", schema=None) as batch_op:
+        batch_op.add_column(sa.Column("is_provisioned", sa.Boolean(), nullable=False, server_default=sa.text("false")))
+
+    # Update existing records to have the default value
+    op.execute("UPDATE alertdeduplicationrule SET is_provisioned = false")
+
+    # Remove the server default (optional, to match schema-only behavior)
+    with op.batch_alter_table("alertdeduplicationrule", schema=None) as batch_op:
+        batch_op.alter_column("is_provisioned", server_default=None)
+    # ### end Alembic commands ###
+
+    # ### end Alembic commands ###
+
+
+def downgrade() -> None:
+        # ### commands auto generated by Alembic - please adjust! ###
+
+    with op.batch_alter_table("alertdeduplicationrule", schema=None) as batch_op:
+        batch_op.drop_column("is_provisioned")
+
+    # ### end Alembic commands ###
diff --git a/keep/parser/parser.py b/keep/parser/parser.py
index d10e4d2ad6..3d5921a98a 100644
--- a/keep/parser/parser.py
+++ b/keep/parser/parser.py
@@ -2,6 +2,7 @@
 import json
 import logging
 import os
+import re
 import typing
 
 import yaml
@@ -252,6 +253,16 @@ def _parse_providers_from_env(self, context_manager: ContextManager):
             (e.g. {"slack-prod": {"authentication": {"webhook_url": "https://hooks.slack.com/services/..."}}})
         """
         providers_json = os.environ.get("KEEP_PROVIDERS")
+
+        # check if env var is absolute or relative path to a providers json file
+        if re.compile(r"^(\/|\.\/|\.\.\/).*\.json$").match(
+            providers_json
+        ):
+            with open(
+                file=providers_json, mode="r", encoding="utf8"
+            ) as file:
+                providers_json = file.read()
+
         if providers_json:
             try:
                 self.logger.debug(
diff --git a/keep/providers/providers_factory.py b/keep/providers/providers_factory.py
index 9e80b4df36..3a2ec4131e 100644
--- a/keep/providers/providers_factory.py
+++ b/keep/providers/providers_factory.py
@@ -625,6 +625,7 @@ def get_default_deduplication_rules() -> list[DeduplicationRuleDto]:
                     full_deduplication=False,
                     # not relevant for default deduplication rules
                     ignore_fields=[],
+                    is_provisioned=False,
                 )
                 default_deduplications.append(deduplication_dto)
 
diff --git a/tests/test_deduplications.py b/tests/deduplication/test_deduplications.py
similarity index 100%
rename from tests/test_deduplications.py
rename to tests/deduplication/test_deduplications.py
diff --git a/tests/deduplication/test_deduplications_provisioning.py b/tests/deduplication/test_deduplications_provisioning.py
new file mode 100644
index 0000000000..75959f2943
--- /dev/null
+++ b/tests/deduplication/test_deduplications_provisioning.py
@@ -0,0 +1,214 @@
+import json
+from uuid import UUID
+import pytest
+from keep.api.alert_deduplicator.deduplication_rules_provisioning import (
+    provision_deduplication_rules_from_env,
+)
+from unittest.mock import patch
+from keep.api.models.db.alert import AlertDeduplicationRule
+from keep.api.models.provider import Provider
+
+
+@pytest.fixture
+def setup(monkeypatch):
+    providers_in_env_var = {
+        "Installed Prometheus provider": {
+            "type": "prometheus",
+            "deduplication_rules": {
+                "provisioned fake existing deduplication rule": {
+                    "description": "new description",
+                    "fingerprint_fields": ["source"],
+                    "full_deduplication": True,
+                    "ignore_fields": ["ignore_field"],
+                }
+            }
+        },
+        "Linked Grafana provider": {
+            "type": "grafana",
+            "deduplication_rules": {
+                "fake new deduplication rule": {
+                    "description": "fake new deduplication rule description",
+                    "fingerprint_fields": ["fingerprint"],
+                    "full_deduplication": False,
+                }
+            }
+        },
+    }
+
+    deduplication_rules_in_db = [
+        AlertDeduplicationRule(
+            id=UUID("f3a2b76c8430491da71684de9cf257ab"),
+            name="provisioned fake existing deduplication rule",
+            description="provisioned fake existing deduplication rule description",
+            provider_id="edc4d65d53204cefb511321be98f748e",
+            provider_type="prometheus",
+            fingerprint_fields=["fingerprint", "source", "service"],
+            full_deduplication=False,
+            is_provisioned=True,
+        ),
+        AlertDeduplicationRule(
+            id=UUID("a5d8f32b6c7049efb913c21da7e845fd"),
+            name="provisioned fake deduplication rule to delete",
+            description="fake new deduplication rule description",
+            provider_id="a1b2c3d4e5f64789ab1234567890abcd",
+            provider_type="grafana",
+            fingerprint_fields=["fingerprint"],
+            full_deduplication=False,
+            is_provisioned=True,
+        ),
+        AlertDeduplicationRule(
+            id=UUID("c7e3d28f95104b6a8f12dc45eb7639fa"),
+            name="not provisioned fake deduplication rule",
+            description="not provisioned fake deduplication rule",
+            provider_id="a1b2c3d4e5f64789ab1234567890abcd",
+            provider_type="grafana",
+            fingerprint_fields=["fingerprint"],
+            full_deduplication=False,
+            is_provisioned=False,
+        ),
+    ]
+    installed_providers = [
+        Provider(
+            id="edc4d65d53204cefb511321be98f748e",
+            name="Installed Prometheus provider",
+            display_name="Installed Prometheus provider",
+            type="prometheus",
+            enabled=True,
+            can_query=True,
+            can_notify=True,
+        ),
+        Provider(
+            id="p2b2c3d4e5f64789ab1234567890abcd",
+            name="Installed Prometheus provider second",
+            display_name="Installed Prometheus provider",
+            type="prometheus",
+            enabled=True,
+            can_query=True,
+            can_notify=True,
+        ),
+    ]
+
+    linked_providers = [
+        Provider(
+            id="a1b2c3d4e5f64789ab1234567890abcd",
+            name="Linked Grafana provider",
+            display_name="Linked Grafana provider",
+            type="grafana",
+            enabled=True,
+            can_query=True,
+            can_notify=True,
+        )
+    ]
+
+    with patch(
+        "keep.api.core.db.get_all_deduplication_rules",
+        return_value=deduplication_rules_in_db,
+    ) as mock_get_all, patch(
+        "keep.api.core.db.delete_deduplication_rule", return_value=None
+    ) as mock_delete, patch(
+        "keep.api.core.db.update_deduplication_rule", return_value=None
+    ) as mock_update, patch(
+        "keep.api.core.db.create_deduplication_rule", return_value=None
+    ) as mock_create, patch(
+        "keep.providers.providers_factory.ProvidersFactory.get_installed_providers",
+        return_value=installed_providers,
+    ) as mock_get_providers, patch(
+        "keep.providers.providers_factory.ProvidersFactory.get_linked_providers",
+        return_value=linked_providers,
+    ) as mock_get_linked_providers:
+
+        fake_tenant_id = "fake_tenant_id"
+        monkeypatch.setenv(
+            "KEEP_PROVIDERS", json.dumps(providers_in_env_var)
+        )
+
+        yield {
+            "mock_get_all": mock_get_all,
+            "mock_delete": mock_delete,
+            "mock_update": mock_update,
+            "mock_create": mock_create,
+            "mock_get_providers": mock_get_providers,
+            "mock_get_linked_providers": mock_get_linked_providers,
+            "fake_tenant_id": fake_tenant_id,
+            "providers_in_env_var": providers_in_env_var,
+            "deduplication_rules_in_db": deduplication_rules_in_db,
+            "linked_providers": linked_providers,
+            "installed_providers": installed_providers,
+        }
+
+
+def test_provisioning_of_new_rule(setup):
+    """
+    Test the provisioning of new deduplication rules from the environment.
+    """
+    provision_deduplication_rules_from_env(setup["fake_tenant_id"])
+    setup["mock_create"].assert_called_once_with(
+        tenant_id=setup["fake_tenant_id"],
+        name="fake new deduplication rule",
+        description="fake new deduplication rule description",
+        provider_id="a1b2c3d4e5f64789ab1234567890abcd",
+        provider_type="grafana",
+        created_by="system",
+        enabled=True,
+        fingerprint_fields=["fingerprint"],
+        full_deduplication=False,
+        ignore_fields=[],
+        priority=0,
+        is_provisioned=True,
+    )
+
+
+def test_provisioning_of_existing_rule(setup):
+    """
+    Test the provisioning of new deduplication rules from the environment.
+    """
+    provision_deduplication_rules_from_env(setup["fake_tenant_id"])
+    setup["mock_update"].assert_called_once_with(
+        tenant_id=setup["fake_tenant_id"],
+        rule_id=str(UUID("f3a2b76c8430491da71684de9cf257ab")),
+        name="provisioned fake existing deduplication rule",
+        description="new description",
+        provider_id="p2b2c3d4e5f64789ab1234567890abcd",
+        provider_type="prometheus",
+        last_updated_by="system",
+        enabled=True,
+        fingerprint_fields=["source"],
+        full_deduplication=True,
+        ignore_fields=["ignore_field"],
+        priority=0,
+    )
+
+
+def test_deletion_of_provisioned_rule_not_in_env(setup):
+    """
+    Test the provisioning of new deduplication rules from the environment.
+    """
+    provision_deduplication_rules_from_env(setup["fake_tenant_id"])
+    setup["mock_delete"].assert_called_once_with(
+        tenant_id=setup["fake_tenant_id"],
+        rule_id=str(UUID("a5d8f32b6c7049efb913c21da7e845fd")),
+    )
+
+def test_not_throwing_error_if_env_var_empty(setup, monkeypatch):
+    monkeypatch.setenv(
+        "KEEP_PROVIDERS", ''
+    )
+    try:
+        provision_deduplication_rules_from_env(setup["fake_tenant_id"])
+    except Exception as e:
+        pytest.fail(f"provision_deduplication_rules_from_env raised an exception: {e}")
+
+def test_not_throwing_error_if_providers_do_not_have_dedup_rules(setup, monkeypatch):
+    providers_in_env_var = {
+        "Installed Prometheus provider": {
+            "type": "prometheus"
+        }
+    }
+
+    monkeypatch.setenv(
+        "KEEP_PROVIDERS", json.dumps(providers_in_env_var)
+    )
+    try:
+        provision_deduplication_rules_from_env(setup["fake_tenant_id"])
+    except Exception as e:
+        pytest.fail(f"provision_deduplication_rules_from_env raised an exception: {e}")    


Code review comments:
diff_hunk: @@ -231,7 +230,8 @@
             "deployment/provision/overview",
             "deployment/provision/provider",
             "deployment/provision/workflow",
-            "deployment/provision/dashboard"
+            "deployment/provision/dashboard",
+            "deployment/provision/deduplication-rule"
path: docs/mint.json
text: @user1:
doesn't exist anymore, right?
```suggestion
```

@author:
Nice catch! Fixed