Repository: PostHog/posthog
Pull request #30280: feat(cdp): add restrictions to hogtransformation creation
Author: login: meikelmosby
Created: 2025-03-21T11:34:12Z | Merged: 2025-03-24T09:57:54Z

## Problem


As we are rolling out hog-transformations out soon (also with the ability to write custom code) we want to take a rather defensive approach to not get taken down. We need to be careful around how many hog transformations run during the ingestion step and we need to make sure to not run into OOM issues when loading all hogfunctions from teams into memory. 

<!-- Who are we building for, what are their needs, why is this important? -->

<!-- Does this fix an issue? Uncomment the line below with the issue ID to automatically close it when merged -->
<!-- Closes #ISSUE_ID -->

## Changes

- API Level restrictions on hog code to not exceed a certain threshold (took our biggest function and multiplied its size by 10) --> this applies also to _**destinations**_
- API Level restrictions on how many hog transformations can be enabled (unlimited creation is possible as long as there is only 20 transformations enabled at a time)
- Added UI validation to prevent creation / updates of hog functions with same size limit as in backend (will also apply to destinations)



(test was done with 20KB which was easier to reach, real limit is 100KB)
![2025-03-21 14 33 17](https://github.com/user-attachments/assets/541bbdaa-fae2-4f08-b1de-094867c3b102)


(limit was done with 10.. real limit is 20)
![Screenshot 2025-03-24 at 10 16 01](https://github.com/user-attachments/assets/65571a69-ba9c-4ffa-b767-298c031a538c)

<!-- If there are frontend changes, please include screenshots. -->
<!-- If a reference design was involved, include a link to the relevant Figma frame! -->

👉 _Stay up-to-date with [PostHog coding conventions](https://posthog.com/docs/contribute/coding-conventions) for a smoother review._

## Does this work well for both Cloud and self-hosted?

<!-- Yes / no / it doesn't have an impact. -->

## How did you test this code?

- added tests 
- Tested locally
- [x] test out locally via api / ui limit of transformation creation
- [x] Test out via api if exceeding hog function code throws error

<!-- Briefly describe the steps you took. -->
<!-- Include automated tests if possible, otherwise describe the manual testing routine. -->



Diff:
diff --git a/frontend/src/scenes/pipeline/hogfunctions/hogFunctionConfigurationLogic.tsx b/frontend/src/scenes/pipeline/hogfunctions/hogFunctionConfigurationLogic.tsx
index f2389e258060..d5410893681d 100644
--- a/frontend/src/scenes/pipeline/hogfunctions/hogFunctionConfigurationLogic.tsx
+++ b/frontend/src/scenes/pipeline/hogfunctions/hogFunctionConfigurationLogic.tsx
@@ -65,6 +65,7 @@ export interface HogFunctionConfigurationLogicProps {
 
 export const EVENT_VOLUME_DAILY_WARNING_THRESHOLD = 1000
 const UNSAVED_CONFIGURATION_TTL = 1000 * 60 * 5
+export const HOG_CODE_SIZE_LIMIT = 100 * 1024 // 100KB to match backend limit
 
 const NEW_FUNCTION_TEMPLATE: HogFunctionTemplateType = {
     id: 'new',
@@ -480,6 +481,19 @@ export const hogFunctionConfigurationLogic = kea<hogFunctionConfigurationLogicTy
                 }
             },
             submit: async (data) => {
+                // Check HOG code size immediately before submission
+                if (data.hog) {
+                    const hogSize = new Blob([data.hog]).size
+                    if (hogSize > HOG_CODE_SIZE_LIMIT) {
+                        lemonToast.error(
+                            `Hog code exceeds maximum size of ${
+                                HOG_CODE_SIZE_LIMIT / 1024
+                            }KB. Please simplify your code or contact support to increase the limit.`
+                        )
+                        return
+                    }
+                }
+
                 const payload: Record<string, any> = sanitizeConfiguration(data)
                 // Only sent on create
                 payload.template_id = props.templateId || values.hogFunction?.template?.id
diff --git a/posthog/api/hog_function.py b/posthog/api/hog_function.py
index c70b5737a5f1..2410dad17f62 100644
--- a/posthog/api/hog_function.py
+++ b/posthog/api/hog_function.py
@@ -43,6 +43,11 @@
 from posthog.plugins.plugin_server_api import create_hog_invocation_test
 from django.conf import settings
 
+# Maximum size of HOG code as a string in bytes (100KB)
+MAX_HOG_CODE_SIZE_BYTES = 100 * 1024
+# Maximum number of transformation functions per team
+MAX_TRANSFORMATIONS_PER_TEAM = 20
+
 logger = structlog.get_logger(__name__)
 
 
@@ -235,11 +240,39 @@ def validate(self, attrs):
             self.context.get("view") and self.context["view"].action == "create"
         )
 
+        # Check for transformation limit per team when the function will be enabled
+        # We allow unlimited creation of disabled transformations as they don't run during ingestion
+        if hog_type == "transformation" and attrs.get("enabled", False):
+            # Don't apply the limit for updates where the function was already enabled
+            apply_limit = is_create or (isinstance(self.instance, HogFunction) and not self.instance.enabled)
+
+            if apply_limit:
+                # Count enabled and non-deleted transformations
+                transformation_count = HogFunction.objects.filter(
+                    team=team, type="transformation", deleted=False, enabled=True
+                ).count()
+
+                if transformation_count >= MAX_TRANSFORMATIONS_PER_TEAM:
+                    raise serializers.ValidationError(
+                        {
+                            "type": f"Maximum of {MAX_TRANSFORMATIONS_PER_TEAM} enabled transformation functions allowed per team. Please contact support if you need this limit increased, or disable some existing transformations."
+                        }
+                    )
+
         if attrs.get("mappings", None) is not None:
             if hog_type not in ["site_destination", "destination"]:
                 raise serializers.ValidationError({"mappings": "Mappings are only allowed for destinations."})
 
         if "hog" in attrs:
+            # First check the raw code size before trying to compile/transpile it
+            hog_code_size = len(attrs["hog"].encode("utf-8"))
+            if hog_code_size > MAX_HOG_CODE_SIZE_BYTES:
+                raise serializers.ValidationError(
+                    {
+                        "hog": f"HOG code exceeds maximum size of {MAX_HOG_CODE_SIZE_BYTES // 1024}KB. Please simplify your code or contact support if you need this limit increased."
+                    }
+                )
+
             if hog_type in TYPES_WITH_JAVASCRIPT_SOURCE:
                 try:
                     # Validate transpilation using the model instance
diff --git a/posthog/api/test/test_hog_function.py b/posthog/api/test/test_hog_function.py
index daa8ab28f319..0f636cfdd54b 100644
--- a/posthog/api/test/test_hog_function.py
+++ b/posthog/api/test/test_hog_function.py
@@ -18,6 +18,7 @@
 from posthog.cdp.templates.webhook.template_webhook import template as template_webhook
 from posthog.cdp.templates.slack.template_slack import template as template_slack
 from posthog.models.team import Team
+from posthog.api.hog_function import MAX_HOG_CODE_SIZE_BYTES, MAX_TRANSFORMATIONS_PER_TEAM
 
 
 EXAMPLE_FULL = {
@@ -1789,3 +1790,135 @@ def test_can_update_with_null_filters(self):
         assert function.filters.get("events", None) is None
         assert function.filters.get("filter_test_accounts", None) is None
         assert function.filters.get("bytecode") is not None
+
+    def test_limits_transformation_functions_per_team(self):
+        """Test that we can create unlimited disabled transformations but only 20 enabled ones"""
+        with override_settings(HOG_TRANSFORMATIONS_CUSTOM_ENABLED_TEAMS=[self.team.id]):
+            # 1. Create several disabled transformations (more than the limit)
+            for i in range(5):
+                response = self.client.post(
+                    f"/api/projects/{self.team.id}/hog_functions/",
+                    data={
+                        "name": f"Disabled Transformation {i}",
+                        "type": "transformation",
+                        "hog": "return event",
+                        "enabled": False,
+                    },
+                )
+                assert response.status_code == status.HTTP_201_CREATED
+
+            # 2. Create enabled transformations up to the limit
+            for i in range(MAX_TRANSFORMATIONS_PER_TEAM):
+                response = self.client.post(
+                    f"/api/projects/{self.team.id}/hog_functions/",
+                    data={
+                        "name": f"Enabled Transformation {i}",
+                        "type": "transformation",
+                        "hog": "return event",
+                        "enabled": True,
+                    },
+                )
+                assert response.status_code == status.HTTP_201_CREATED
+
+            # 3. Verify we hit the limit when trying to create one more enabled transformation
+            response = self.client.post(
+                f"/api/projects/{self.team.id}/hog_functions/",
+                data={
+                    "name": "One Too Many",
+                    "type": "transformation",
+                    "hog": "return event",
+                    "enabled": True,
+                },
+            )
+            assert response.status_code == status.HTTP_400_BAD_REQUEST
+            assert "Maximum of 20 enabled transformation functions" in response.json()["detail"]
+
+            # 4. Verify we can still create disabled transformations when at the limit
+            response = self.client.post(
+                f"/api/projects/{self.team.id}/hog_functions/",
+                data={
+                    "name": "Another Disabled",
+                    "type": "transformation",
+                    "hog": "return event",
+                    "enabled": False,
+                },
+            )
+            assert response.status_code == status.HTTP_201_CREATED
+
+            # 5. Test that we can enable after deleting an enabled one
+            # First delete an enabled transformation
+            enabled_transformation = HogFunction.objects.filter(
+                team=self.team, type="transformation", deleted=False, enabled=True
+            ).first()
+
+            assert enabled_transformation is not None, "No enabled transformation found to delete"
+            self.client.patch(
+                f"/api/projects/{self.team.id}/hog_functions/{enabled_transformation.id}/",
+                data={"deleted": True},
+            )
+
+            # Then enable a disabled transformation
+            disabled_transformation = HogFunction.objects.filter(
+                team=self.team, type="transformation", deleted=False, enabled=False
+            ).first()
+
+            assert disabled_transformation is not None, "No disabled transformation found to enable"
+            response = self.client.patch(
+                f"/api/projects/{self.team.id}/hog_functions/{disabled_transformation.id}/",
+                data={"enabled": True},
+            )
+            assert response.status_code == status.HTTP_200_OK
+
+    def test_validates_raw_hog_code_size(self):
+        with override_settings(HOG_TRANSFORMATIONS_CUSTOM_ENABLED_TEAMS=[self.team.id]):
+            """Test that we validate the raw HOG code size before compiling it."""
+            # Generate a large HOG code string that exceeds the maximum allowed size
+            large_hog_code = "return " + "x" * (MAX_HOG_CODE_SIZE_BYTES + 1000)
+
+            # Try to create a function with HOG code exceeding the size limit
+            # No need to mock compile_hog as we're checking the string size directly
+            response = self.client.post(
+                f"/api/projects/{self.team.id}/hog_functions/",
+                data={
+                    "name": "Large HOG Code Function",
+                    "type": "transformation",
+                    "hog": large_hog_code,
+                },
+            )
+
+            # Verify the creation was rejected with the correct error
+            assert response.status_code == status.HTTP_400_BAD_REQUEST, response.json()
+            assert "HOG code exceeds maximum size" in response.json()["detail"]
+            assert f"{MAX_HOG_CODE_SIZE_BYTES // 1024}KB" in response.json()["detail"]
+
+    def test_validates_raw_hog_code_size_during_update(self):
+        with override_settings(HOG_TRANSFORMATIONS_CUSTOM_ENABLED_TEAMS=[self.team.id]):
+            """Test that we validate the raw HOG code size when updating an existing function."""
+            # First create a hog function with small, valid code
+            response = self.client.post(
+                f"/api/projects/{self.team.id}/hog_functions/",
+                data={
+                    "name": "Valid HOG Code Function",
+                    "type": "transformation",
+                    "hog": "return event",
+                },
+            )
+
+            assert response.status_code == status.HTTP_201_CREATED, response.json()
+            function_id = response.json()["id"]
+
+            # Generate a large HOG code string for the update that exceeds the limit
+            large_hog_code = "return " + "x" * (MAX_HOG_CODE_SIZE_BYTES + 1000)
+
+            # Update the function with large HOG code
+            update_response = self.client.patch(
+                f"/api/projects/{self.team.id}/hog_functions/{function_id}/",
+                data={
+                    "hog": large_hog_code,
+                },
+            )
+
+            # Verify the update was rejected with the correct error
+            assert update_response.status_code == status.HTTP_400_BAD_REQUEST, update_response.json()
+            assert "HOG code exceeds maximum size" in update_response.json()["detail"]
+            assert f"{MAX_HOG_CODE_SIZE_BYTES // 1024}KB" in update_response.json()["detail"]


Reviews:
user: login: greptile-apps[bot]
body: 
## PR Summary

Implemented API-level restrictions for HogFunction creation to enhance system stability and prevent resource abuse in PostHog's CDP functionality.

- Added `MAX_TRANSFORMATIONS_PER_TEAM` constant (20) in `/posthog/api/test/test_hog_function.py` to limit enabled transformations per team
- Introduced `MAX_BYTECODE_SIZE_BYTES` (70KB) limit in `/posthog/api/hog_function.py` to prevent excessive function sizes
- Added validation logic in `/posthog/api/hog_function.py` to enforce these limits with clear error messages directing users to contact support
- Implemented comprehensive test coverage in `/posthog/api/test/test_hog_function.py` for both bytecode size and transformation count restrictions

<sub>2 file(s) reviewed, 2 comment(s)</sub>
<sub>[Edit PR Review Bot Settings](https://app.greptile.com/review/github) | [Greptile](https://greptile.com?utm_source=greptile_expert&utm_medium=github&utm_campaign=code_reviews&utm_content=posthog_30280)</sub>
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-21T11:35:13Z
user: login: benjackwhite
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-03-21T13:59:09Z

Review comments:
diff_hunk: @@ -1789,3 +1790,165 @@ def test_can_update_with_null_filters(self):
         assert function.filters.get("events", None) is None
         assert function.filters.get("filter_test_accounts", None) is None
         assert function.filters.get("bytecode") is not None
+
+    def test_validates_bytecode_size(self):
+        # Mock compile_hog to return a large bytecode array
+        large_bytecode = ["_H", 1] + ["x" for _ in range(MAX_BYTECODE_SIZE_BYTES)]
path: posthog/api/test/test_hog_function.py
user: login: greptile-apps[bot]
body: style: Consider using a constant for the test bytecode size rather than creating a large array inline
created_at: 2025-03-21T11:34:57Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1789,3 +1790,165 @@ def test_can_update_with_null_filters(self):
         assert function.filters.get("events", None) is None
         assert function.filters.get("filter_test_accounts", None) is None
         assert function.filters.get("bytecode") is not None
+
+    def test_validates_bytecode_size(self):
+        # Mock compile_hog to return a large bytecode array
+        large_bytecode = ["_H", 1] + ["x" for _ in range(MAX_BYTECODE_SIZE_BYTES)]
+
+        with patch("posthog.api.hog_function.compile_hog") as mock_compile_hog:
+            # For the first test, return bytecode that's too large
+            with override_settings(HOG_TRANSFORMATIONS_CUSTOM_ENABLED_TEAMS=[self.team.id]):
+                mock_compile_hog.return_value = large_bytecode
+
+                response = self.client.post(
+                    f"/api/projects/{self.team.id}/hog_functions/",
+                    data={
+                        "name": "Large Bytecode Function",
+                        "type": "transformation",
+                        "hog": "return event",
+                    },
+                )
+
+                assert response.status_code == status.HTTP_400_BAD_REQUEST, response.json()
+                assert response.json() == {
+                    "type": "validation_error",
+                    "code": "invalid_input",
+                    "detail": f"Compiled bytecode exceeds maximum size of {MAX_BYTECODE_SIZE_BYTES // 1024}KB. Please contact support if you need this limit increased.",
+                    "attr": "hog",
+                }
+
+                # For the second test, return bytecode that's within the limit
+                valid_bytecode = ["_H", 1, "x", "y", "z"]
+                mock_compile_hog.return_value = valid_bytecode
+
+                response = self.client.post(
+                    f"/api/projects/{self.team.id}/hog_functions/",
+                    data={
+                        "name": "Valid Bytecode Function",
+                        "type": "transformation",
+                        "hog": "return event",
+                    },
+                )
+
+                assert response.status_code == status.HTTP_201_CREATED, response.json()
+                assert response.json()["bytecode"] == valid_bytecode
+
+    def test_limits_transformation_functions_per_team(self):
+        """Test that we can create unlimited disabled transformations but only 20 enabled ones"""
+        with override_settings(HOG_TRANSFORMATIONS_CUSTOM_ENABLED_TEAMS=[self.team.id]):
+            # 1. Create several disabled transformations (more than the limit)
+            for i in range(5):
+                response = self.client.post(
+                    f"/api/projects/{self.team.id}/hog_functions/",
+                    data={
+                        "name": f"Disabled Transformation {i}",
+                        "type": "transformation",
+                        "hog": "return event",
+                        "enabled": False,
+                    },
+                )
+                assert response.status_code == status.HTTP_201_CREATED
+
+            # 2. Create enabled transformations up to the limit
+            for i in range(MAX_TRANSFORMATIONS_PER_TEAM):
+                response = self.client.post(
+                    f"/api/projects/{self.team.id}/hog_functions/",
+                    data={
+                        "name": f"Enabled Transformation {i}",
+                        "type": "transformation",
+                        "hog": "return event",
+                        "enabled": True,
+                    },
+                )
+                assert response.status_code == status.HTTP_201_CREATED
+
+            # 3. Verify we hit the limit when trying to create one more enabled transformation
+            response = self.client.post(
+                f"/api/projects/{self.team.id}/hog_functions/",
+                data={
+                    "name": "One Too Many",
+                    "type": "transformation",
+                    "hog": "return event",
+                    "enabled": True,
+                },
+            )
+            assert response.status_code == status.HTTP_400_BAD_REQUEST
+            assert "Maximum of 20 enabled transformation functions" in response.json()["detail"]
+
+            # 4. Verify we can still create disabled transformations when at the limit
+            response = self.client.post(
+                f"/api/projects/{self.team.id}/hog_functions/",
+                data={
+                    "name": "Another Disabled",
+                    "type": "transformation",
+                    "hog": "return event",
+                    "enabled": False,
+                },
+            )
+            assert response.status_code == status.HTTP_201_CREATED
+
+            # 5. Test that we can enable after deleting an enabled one
+            # First delete an enabled transformation
+            enabled_transformation = HogFunction.objects.filter(
+                team=self.team, type="transformation", deleted=False, enabled=True
+            ).first()
+
+            self.client.patch(
+                f"/api/projects/{self.team.id}/hog_functions/{enabled_transformation.id}/",
+                data={"deleted": True},
+            )
+
+            # Then enable a disabled transformation
+            disabled_transformation = HogFunction.objects.filter(
+                team=self.team, type="transformation", deleted=False, enabled=False
+            ).first()
+
+            response = self.client.patch(
+                f"/api/projects/{self.team.id}/hog_functions/{disabled_transformation.id}/",
+                data={"enabled": True},
+            )
+            assert response.status_code == status.HTTP_200_OK
+
+    def test_validates_bytecode_size_during_update(self):
+        """Test that we validate bytecode size when updating an existing function."""
+        # First create a hog function with small, valid bytecode
+        with override_settings(HOG_TRANSFORMATIONS_CUSTOM_ENABLED_TEAMS=[self.team.id]):
+            small_bytecode = ["_H", 1, "x", "y", "z"]
+
+            with patch("posthog.api.hog_function.compile_hog") as mock_compile_hog:
+                # Return small bytecode for creation
+                mock_compile_hog.return_value = small_bytecode
+
+                # Create the function
+                response = self.client.post(
+                    f"/api/projects/{self.team.id}/hog_functions/",
+                    data={
+                        "name": "Valid Bytecode Function",
+                        "type": "transformation",
+                        "hog": "return event",
+                    },
+                )
+
+                assert response.status_code == status.HTTP_201_CREATED, response.json()
+                function_id = response.json()["id"]
+
+                # Now attempt to update with a large bytecode that exceeds the limit
+                large_bytecode = ["_H", 1] + ["x" for _ in range(MAX_BYTECODE_SIZE_BYTES)]
+                mock_compile_hog.return_value = large_bytecode
path: posthog/api/test/test_hog_function.py
user: login: greptile-apps[bot]
body: style: Duplicate code from earlier test - consider extracting large_bytecode creation to a helper function
created_at: 2025-03-21T11:34:58Z
author_association: CONTRIBUTOR

PR comments:
user: login: posthog-bot
created_at: 2025-03-21T11:34:26Z
body: Hey @meikelmosby! 👋
This pull request seems to contain no description. Please add useful context, rationale, and/or any other information that will help make sense of this change now and in the distant Mars-based future.
author_association: CONTRIBUTOR
user: login: github-actions[bot]
created_at: 2025-03-21T13:46:08Z
body: **Size Change:** +7 B (0%) 

**Total Size:** 9.75 MB



<details><summary>ℹ️ <strong>View Unchanged</strong></summary>

| Filename | Size | Change |
| :--- | :---: | :---: |
| `frontend/dist/toolbar.js` | 9.75 MB | +7 B (0%) |

</details>



<a href="https://github.com/preactjs/compressed-size-action"><sub>compressed-size-action</sub></a>
author_association: CONTRIBUTOR