Repository: apache/airflow
Pull request: Configurable resources for git-sync sidecar
Add configurable resources for the `git-sync` sidecar.

closes: #16056



Problem statement:
[Helm] Resources for the git-sync sidecar
**Description**

It would be nice to be able to specify resources for the `git-sync` sidecar in the helm chart values.

**Use case / motivation**

I don't want to use keda for autoscaling and would like to setup a HPA myself. However this is currently not possible since it is not possible to specify resources for the `git-sync` sidecar.

**Are you willing to submit a PR?**

Yes, I am willing to submit a PR.

**Related Issues**

Not that I know of.

Hints from issue discussion:
Thanks for opening your first issue here! Be sure to follow the issue template!

Go for it @CyberHippo

Commit message: Feat: configurable resources for git-sync sidecar

Patch to review:
diff --git a/chart/templates/_helpers.yaml b/chart/templates/_helpers.yaml
index d8bae0bfe0135..ab1553d9f66eb 100644
--- a/chart/templates/_helpers.yaml
+++ b/chart/templates/_helpers.yaml
@@ -193,6 +193,7 @@ If release name contains chart name it will be used as a full name.
     - name: {{ $config.name }}
       value: {{ $config.value | quote }}
     {{- end }}
+  resources: {{ toYaml .Values.dags.gitSync.resources | nindent 6 }}
   volumeMounts:
   - name: dags
     mountPath: /git
diff --git a/chart/tests/test_git_sync_scheduler.py b/chart/tests/test_git_sync_scheduler.py
index 9911946fd7213..8079571bc0580 100644
--- a/chart/tests/test_git_sync_scheduler.py
+++ b/chart/tests/test_git_sync_scheduler.py
@@ -98,6 +98,7 @@ def test_validate_the_git_sync_container_spec(self):
                 {"name": "GIT_SYNC_MAX_SYNC_FAILURES", "value": "70"},
             ],
             "volumeMounts": [{"mountPath": "/git", "name": "dags"}],
+            "resources": {},
         } == jmespath.search("spec.template.spec.containers[1]", docs[0])
 
     def test_validate_if_ssh_params_are_added(self):
@@ -230,3 +231,24 @@ def test_should_add_env(self):
         assert {"name": "FOO", "value": "bar"} in jmespath.search(
             "spec.template.spec.containers[1].env", docs[0]
         )
+
+    def test_resources_are_configurable(self):
+        docs = render_chart(
+            values={
+                "dags": {
+                    "gitSync": {
+                        "enabled": True,
+                        "resources": {
+                            "limits": {"cpu": "200m", 'memory': "128Mi"},
+                            "requests": {"cpu": "300m", 'memory': "169Mi"},
+                        },
+                    },
+                },
+            },
+            show_only=["templates/scheduler/scheduler-deployment.yaml"],
+        )
+        assert "128Mi" == jmespath.search("spec.template.spec.containers[1].resources.limits.memory", docs[0])
+        assert "169Mi" == jmespath.search(
+            "spec.template.spec.containers[1].resources.requests.memory", docs[0]
+        )
+        assert "300m" == jmespath.search("spec.template.spec.containers[1].resources.requests.cpu", docs[0])
diff --git a/chart/tests/test_git_sync_webserver.py b/chart/tests/test_git_sync_webserver.py
index 4c6f544d1ff28..9c12ba01af837 100644
--- a/chart/tests/test_git_sync_webserver.py
+++ b/chart/tests/test_git_sync_webserver.py
@@ -148,3 +148,29 @@ def test_should_add_env(self):
         assert {"name": "FOO", "value": "bar"} in jmespath.search(
             "spec.template.spec.containers[1].env", docs[0]
         )
+
+    def test_resources_are_configurable(self):
+        docs = render_chart(
+            values={
+                "airflowVersion": "1.10.14",
+                "dags": {
+                    "gitSync": {
+                        "enabled": True,
+                        "resources": {
+                            "limits": {"cpu": "200m", 'memory': "128Mi"},
+                            "requests": {"cpu": "300m", 'memory': "169Mi"},
+                        },
+                    },
+                },
+            },
+            show_only=["templates/webserver/webserver-deployment.yaml"],
+        )
+        assert "128Mi" == jmespath.search(
+            "spec.template.spec.containers[1].resources.limits.memory", docs[0]
+        )
+        assert "169Mi" == jmespath.search(
+            "spec.template.spec.containers[1].resources.requests.memory", docs[0]
+        )
+        assert "300m" == jmespath.search(
+            "spec.template.spec.containers[1].resources.requests.cpu", docs[0]
+        )
diff --git a/chart/tests/test_git_sync_worker.py b/chart/tests/test_git_sync_worker.py
index d7e9b4407c6c8..9ab1ef2b75db2 100644
--- a/chart/tests/test_git_sync_worker.py
+++ b/chart/tests/test_git_sync_worker.py
@@ -91,3 +91,24 @@ def test_should_add_env(self):
         assert {"name": "FOO", "value": "bar"} in jmespath.search(
             "spec.template.spec.containers[1].env", docs[0]
         )
+
+    def test_resources_are_configurable(self):
+        docs = render_chart(
+            values={
+                "dags": {
+                    "gitSync": {
+                        "enabled": True,
+                        "resources": {
+                            "limits": {"cpu": "200m", 'memory': "128Mi"},
+                            "requests": {"cpu": "300m", 'memory': "169Mi"},
+                        },
+                    },
+                },
+            },
+            show_only=["templates/workers/worker-deployment.yaml"],
+        )
+        assert "128Mi" == jmespath.search("spec.template.spec.containers[1].resources.limits.memory", docs[0])
+        assert "169Mi" == jmespath.search(
+            "spec.template.spec.containers[1].resources.requests.memory", docs[0]
+        )
+        assert "300m" == jmespath.search("spec.template.spec.containers[1].resources.requests.cpu", docs[0])
diff --git a/chart/tests/test_pod_template_file.py b/chart/tests/test_pod_template_file.py
index 44b4fa8ef6c5b..c6d57bff8df0c 100644
--- a/chart/tests/test_pod_template_file.py
+++ b/chart/tests/test_pod_template_file.py
@@ -102,6 +102,7 @@ def test_should_add_an_init_container_if_git_sync_is_true(self):
                 {"name": "GIT_SYNC_ONE_TIME", "value": "true"},
             ],
             "volumeMounts": [{"mountPath": "/git", "name": "dags"}],
+            "resources": {},
         } == jmespath.search("spec.initContainers[0]", docs[0])
 
     def test_should_not_add_init_container_if_dag_persistence_is_true(self):
diff --git a/chart/values.schema.json b/chart/values.schema.json
index 8b34093dbba6f..890dcb9c07e7d 100644
--- a/chart/values.schema.json
+++ b/chart/values.schema.json
@@ -2454,6 +2454,23 @@
                                     "value": "60"
                                 }
                             ]
+                        },
+                        "resources": {
+                            "description": "Resources on workers git-sync sidecar",
+                            "type": "object",
+                            "default": {},
+                            "examples": [
+                                {
+                                    "limits": {
+                                        "cpu": "100m",
+                                        "memory": "128Mi"
+                                    },
+                                    "requests": {
+                                        "cpu": "100m",
+                                        "memory": "128Mi"
+                                    }
+                                }
+                            ]
                         }
                     }
                 }
diff --git a/chart/values.yaml b/chart/values.yaml
index 9ef17ae6a8762..dd068c592dd15 100644
--- a/chart/values.yaml
+++ b/chart/values.yaml
@@ -1015,6 +1015,13 @@ dags:
     uid: 65533
     extraVolumeMounts: []
     env: []
+    resources: {}
+    #  limits:
+    #   cpu: 100m
+    #   memory: 128Mi
+    #  requests:
+    #   cpu: 100m
+    #   memory: 128Mi
 
 logs:
   persistence:


Code review comments:
diff_hunk: @@ -148,3 +148,29 @@ def test_should_add_env(self):
         assert {"name": "FOO", "value": "bar"} in jmespath.search(
             "spec.template.spec.containers[1].env", docs[0]
         )
+
+    def test_resources_are_configurable(self):
+        docs = render_chart(
+            values={
+                "airflowVersion": "1.10.14",
+                "dags": {
+                    "gitSync": {
+                        "enabled": True,
+                        "resources": {
+                            "limits": {"cpu": "200m", 'memory': "128Mi"},
+                            "requests": {"cpu": "300m", 'memory': "169Mi"},
+                        },
+                    },
+                },
+            },
+            show_only=["templates/webserver/webserver-deployment.yaml"],
+        )
+        assert "128Mi" == jmespath.search(
+            "spec.template.spec.containers[1].resources.limits.memory", docs[0]
+        )
+        assert "169Mi" == jmespath.search(
+            "spec.template.spec.containers[1].resources.requests.memory", docs[0]
+        )
+        assert "300m" == jmespath.search(
+            "spec.template.spec.containers[1].resources.requests.cpu", docs[0]
+        )
path: chart/tests/test_git_sync_webserver.py
text: @user1:
```suggestion
        assert "300m" == jmespath.search("spec.template.spec.containers[1].resources.requests.cpu", docs[0])
```