Repository: apache/airflow

Issue:
Use private docker repository with K8S operator and XCOM sidecar container
**Use private docker repository with K8S operator and XCOM sidecar container**

An extra parameter to KubernetesPodOperator: docker_repository, this allows to specify the repository where the sidecar container is located

**My company force docker proxy usage for K8S**

I need to use my company docker repository, images that are not proxifed by the company docker repository are not allowed



Discussion:
Thanks for opening your first issue here! Be sure to follow the issue template!

would you like to add this feature @mickaelgervais ? I am happy to review it 
Hi @potiuk ,

I've submitted a PR, do nopt hesitate to add comment or if I've implemented this badly. 
The sidecar container declaration is low level in K8S operator so I've tried to not add complexity.

Regards
Hi @mickaelgervais 
To understand your issue better, are you having trouble turning on xcom because 'alpine' give you 403 error? Because in your description, you are not able to use private registry, but in my case, I can pass the whole string to the image in my KubernetesPodOperator like below. I just have problem with the xcom-sidecar, because class PodDefaults hard coded image name to be 'alpine'.
`image="<private-registry>/<dir>/<dir>/<image>:<tag>"`

I have left some comments in your pr for adding a explicit image tag, we have something like private-registry/alpine:3.10, and would like to add a variable for image tags.

Hi @potiuk 

Is this feature going live in 1.10.13 or later versions?
A work around would be to add this code block in the /home/airflow/airflow_local_settings.py
PodDefaults.SIDECAR_CONTAINER = k8s.V1Container(
    name=PodDefaults.SIDECAR_CONTAINER_NAME,
    command=['sh', '-c', PodDefaults.XCOM_CMD],
    image='<local private repository image>',
    volume_mounts=[PodDefaults.VOLUME_MOUNT],
    resources=k8s.V1ResourceRequirements(
        requests={
            "cpu": "10m",
            "memory":"32Mi"
        },
        limits={
            "cpu": "0.25",
            "memory":"32Mi"
        }
    ),
)
Docker Inc started image pull rate limiting around October 30th 2020 (https://www.docker.com/blog/checking-your-current-docker-pull-rate-limits-and-status/). As a result, all KubernetesPodPerator tasks with XCOM started to fail in large deployment. Can we have the fix in Airflow 1.10 branch as well?

Based on the severity of the failure, the bug priority should be raised higher.

```
  - image: alpine
    imageID: ""
    lastState: {}
    name: airflow-xcom-sidecar
    ready: false
    restartCount: 0
    started: false
    state:
      waiting:
        message: 'rpc error: code = Unknown desc = Error response from daemon: toomanyrequests:
          You have reached your pull rate limit. You may increase the limit by authenticating
          and upgrading: https://www.docker.com/increase-rate-limit'
        reason: ErrImagePull
```
Is there a reason why this has not been implemented yet?
> Is there a reason why this has not been implemented yet?

It was waiting for someone like you to take that over and complete it possibly? Airflow has > 1700 contributors and you can become one if you want some features to be implemented faster :)
Feel free to take it over and contribute! we even have Contributors workshops  (3 of them till the end of the year). On Airflow Slack there is a #contributors-workshop dedicated channel for it! Anyone is welcome!
sure, I think it's covered in a different feature request, let me find out
and update the comments.

Can you let me know when are the workshops and how do I attend them?

Thanks,
Zhicong

On Wed, Sep 22, 2021 at 2:22 PM Jarek Potiuk ***@***.***>
wrote:

> Feel free to take it over and contribute! we even have Contributors
> workshops (3 of them till the end of the year). On Airflow Slack there is a
> #contributors-workshop dedicated channel for it! Anyone is welcome!
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/airflow/issues/10605#issuecomment-925246273>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AB52K2YS5FVVOH7QE55OSTTUDIUHXANCNFSM4QNED7HQ>
> .
> Triage notifications on the go with GitHub Mobile for iOS
> <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
> or Android
> <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
>
>

Here is the workaround (tested with Airflow 2.1.4) if you are using airflow helm chart:


values.yaml:

```
airflowLocalSettings: |
  from airflow.providers.cncf.kubernetes.utils.xcom_sidecar import PodDefaults
  PodDefaults.SIDECAR_CONTAINER.image = private.registry.com/alpine
```
> Can you let me know when are the workshops and how do I attend them?

Just join the #contributors-workshop channel - we announce the workshops there.
> airflowLocalSettings: |
>   from airflow.providers.cncf.kubernetes.utils.xcom_sidecar import PodDefaults
>   PodDefaults.SIDECAR_CONTAINER.image = private.registry.com/alpine

@fritzb

I tried something like this in the past and it didnt work. However this worked for me in 2.2.0, so thank you!
> > airflowLocalSettings: |
> > from airflow.providers.cncf.kubernetes.utils.xcom_sidecar import PodDefaults
> > PodDefaults.SIDECAR_CONTAINER.image = private.registry.com/alpine
> 
> @fritzb
> 
> I tried something like this in the past and it didnt work. However this worked for me in 2.2.0, so thank you!

I was able to get my xcom to use a non-DockerHub repo also by creating a `airflow_local_settings.py` file in the root/config folder, due to this yaml:
https://github.com/apache/airflow/blob/73044af1c6908d1b075bd0127bc56c1db82b564b/chart/templates/_helpers.yaml#L416.

I then simply overwrote with the code:
```
from airflow.providers.cncf.kubernetes.utils.xcom_sidecar import PodDefaults
PodDefaults.SIDECAR_CONTAINER.image = "docker.very.private.repo.net/alpine:3.10"
```
You can verify the custom `airflow_local_settings.py` file is loaded because any code written in that file will be executed by teach task instance, i.e., those are the logs to check.  That's really the only annoying/hard part, is just knowing whether your instance is actually executing/including the local_settings file.

Also note that this works even as early as at least 2.1.0 for those of you who are having issues with 2.2.0 :)
Hi @ZackingIt , could I know how do you overrote the code? Do you go to pod and change the  airflow_local_settings.py file in root/config folder or you add it in values.yaml file as below:

Your code
```
from airflow.providers.cncf.kubernetes.utils.xcom_sidecar import PodDefaults
PodDefaults.SIDECAR_CONTAINER.image = "docker.very.private.repo.net/alpine:3.10"
```

in values.yaml
```
airflow
    localSettings:
      ## the full content of the `airflow_local_settings.py` file (as a string)
      stringOverride: |
        # use a custom `xcom_sidecar` image for KubernetesPodOperator()
        from airflow.kubernetes.pod_generator import PodDefaults
        PodDefaults.SIDECAR_CONTAINER.image = "private_artifactory/alpine"
```





@dzungdev 
 I created an actual file, I did *not* overwrite a variable in the `values.yaml` i.e., part of my Docker build involves:
`cp /airflow-config/config/airflow_local_settings.py ${AIRFLOW_HOME}/config/airflow_local_settings.py`

When I say "due to this yaml" I mean that that yaml tells us that this is already a location,`${AIRFLOW_HOME}/config` is already one of the paths being read by default, as I did not need to explicitly set a PYTHONPATH environment variable to read that location.
@ZackingIt 
Where exactly did you put the configuration? When I put the `PodDefaults.SIDECAR_CONTAINER.image` (as mentioned several times here and everywhere else) into the `values.yaml` it is just used by the helm chart. But I can't figure out where to put the `airflow_local_settings.py` to configure the worker pod image pulled by the `KubernetesExecutor`. I even can't build a Docker image with Airflow 2.1.4 which is configured as needed.

My current Dockerfile looks like the following:
```Dockerfile
FROM apache/airflow:2.1.4-python3.8@sha256:ae98e5e9eba596e24bce820e5f3a109274b82cd347bb9fdfca1e9d3f0dda4bb0

COPY --chown=airflow:root airflow_local_settings.py /opt/airflow/config/airflow_local_settings.py

USER airflow
```
But as soon as the `KubernetesExecutor` is pulling my image (because I configured it in the `values.yaml`) it runs into the Docker pull limit very easy, because I can't configure the image correctly.
We are facing the same issue, but in our case the error is related the absent of an option to define resources to the sidecar container. Our kubernetes cluster has this resource enforcement active and them even trying to set the resouces by airflow_local_settings.py, pod_template_file, it sounds like the code is ignoring these options.

Resolving patch:
diff --git a/airflow/providers/cncf/kubernetes/hooks/kubernetes.py b/airflow/providers/cncf/kubernetes/hooks/kubernetes.py
--- a/airflow/providers/cncf/kubernetes/hooks/kubernetes.py
+++ b/airflow/providers/cncf/kubernetes/hooks/kubernetes.py
@@ -91,6 +91,9 @@ def get_connection_form_widgets() -> dict[str, Any]:
             "cluster_context": StringField(lazy_gettext("Cluster context"), widget=BS3TextFieldWidget()),
             "disable_verify_ssl": BooleanField(lazy_gettext("Disable SSL")),
             "disable_tcp_keepalive": BooleanField(lazy_gettext("Disable TCP keepalive")),
+            "xcom_sidecar_container_image": StringField(
+                lazy_gettext("XCom sidecar image"), widget=BS3TextFieldWidget()
+            ),
         }
 
     @staticmethod
@@ -347,6 +350,10 @@ def _get_namespace(self) -> str | None:
             return self._get_field("namespace")
         return None
 
+    def get_xcom_sidecar_container_image(self):
+        """Returns the xcom sidecar image that defined in the connection"""
+        return self._get_field("xcom_sidecar_container_image")
+
     def get_pod_log_stream(
         self,
         pod_name: str,
diff --git a/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py b/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py
--- a/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py
+++ b/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py
@@ -591,7 +591,9 @@ def build_pod_request_obj(self, context: Context | None = None) -> k8s.V1Pod:
             pod = secret.attach_to_pod(pod)
         if self.do_xcom_push:
             self.log.debug("Adding xcom sidecar to task %s", self.task_id)
-            pod = xcom_sidecar.add_xcom_sidecar(pod)
+            pod = xcom_sidecar.add_xcom_sidecar(
+                pod, sidecar_container_image=self.hook.get_xcom_sidecar_container_image()
+            )
 
         labels = self._get_ti_pod_labels(context)
         self.log.info("Building pod %s with labels: %s", pod.metadata.name, labels)
diff --git a/airflow/providers/cncf/kubernetes/utils/xcom_sidecar.py b/airflow/providers/cncf/kubernetes/utils/xcom_sidecar.py
--- a/airflow/providers/cncf/kubernetes/utils/xcom_sidecar.py
+++ b/airflow/providers/cncf/kubernetes/utils/xcom_sidecar.py
@@ -47,13 +47,15 @@ class PodDefaults:
     )
 
 
-def add_xcom_sidecar(pod: k8s.V1Pod) -> k8s.V1Pod:
+def add_xcom_sidecar(pod: k8s.V1Pod, *, sidecar_container_image=None) -> k8s.V1Pod:
     """Adds sidecar"""
     pod_cp = copy.deepcopy(pod)
     pod_cp.spec.volumes = pod.spec.volumes or []
     pod_cp.spec.volumes.insert(0, PodDefaults.VOLUME)
     pod_cp.spec.containers[0].volume_mounts = pod_cp.spec.containers[0].volume_mounts or []
     pod_cp.spec.containers[0].volume_mounts.insert(0, PodDefaults.VOLUME_MOUNT)
-    pod_cp.spec.containers.append(PodDefaults.SIDECAR_CONTAINER)
+    sidecar = copy.deepcopy(PodDefaults.SIDECAR_CONTAINER)
+    sidecar.image = sidecar_container_image or PodDefaults.SIDECAR_CONTAINER.image
+    pod_cp.spec.containers.append(sidecar)
 
     return pod_cp


Test patch:
diff --git a/kubernetes_tests/test_kubernetes_pod_operator.py b/kubernetes_tests/test_kubernetes_pod_operator.py
--- a/kubernetes_tests/test_kubernetes_pod_operator.py
+++ b/kubernetes_tests/test_kubernetes_pod_operator.py
@@ -922,6 +922,7 @@ def test_pod_template_file(
         # todo: This isn't really a system test
         await_xcom_sidecar_container_start_mock.return_value = None
         hook_mock.return_value.is_in_cluster = False
+        hook_mock.return_value.get_xcom_sidecar_container_image.return_value = None
         extract_xcom_mock.return_value = "{}"
         path = sys.path[0] + "/tests/kubernetes/pod.yaml"
         k = KubernetesPodOperator(
diff --git a/tests/providers/cncf/kubernetes/hooks/test_kubernetes.py b/tests/providers/cncf/kubernetes/hooks/test_kubernetes.py
--- a/tests/providers/cncf/kubernetes/hooks/test_kubernetes.py
+++ b/tests/providers/cncf/kubernetes/hooks/test_kubernetes.py
@@ -62,6 +62,8 @@ def setup_class(cls) -> None:
             ("disable_verify_ssl_empty", {"disable_verify_ssl": ""}),
             ("disable_tcp_keepalive", {"disable_tcp_keepalive": True}),
             ("disable_tcp_keepalive_empty", {"disable_tcp_keepalive": ""}),
+            ("sidecar_container_image", {"xcom_sidecar_container_image": "private.repo.com/alpine:3.16"}),
+            ("sidecar_container_image_empty", {"xcom_sidecar_container_image": ""}),
         ]:
             db.merge_conn(Connection(conn_type="kubernetes", conn_id=conn_id, extra=json.dumps(extra)))
 
@@ -316,6 +318,17 @@ def test_get_namespace(self, conn_id, expected):
                 "and rename _get_namespace to get_namespace."
             )
 
+    @pytest.mark.parametrize(
+        "conn_id, expected",
+        (
+            pytest.param("sidecar_container_image", "private.repo.com/alpine:3.16", id="sidecar-with-image"),
+            pytest.param("sidecar_container_image_empty", None, id="sidecar-without-image"),
+        ),
+    )
+    def test_get_xcom_sidecar_container_image(self, conn_id, expected):
+        hook = KubernetesHook(conn_id=conn_id)
+        assert hook.get_xcom_sidecar_container_image() == expected
+
     @patch("kubernetes.config.kube_config.KubeConfigLoader")
     @patch("kubernetes.config.kube_config.KubeConfigMerger")
     def test_client_types(self, mock_kube_config_merger, mock_kube_config_loader):
diff --git a/tests/providers/cncf/kubernetes/operators/test_kubernetes_pod.py b/tests/providers/cncf/kubernetes/operators/test_kubernetes_pod.py
--- a/tests/providers/cncf/kubernetes/operators/test_kubernetes_pod.py
+++ b/tests/providers/cncf/kubernetes/operators/test_kubernetes_pod.py
@@ -413,6 +413,28 @@ def test_omitted_namespace_no_conn_not_in_k8s(self, mock_find, mock_path):
         )
         mock_find.assert_called_once_with("default", context=context)
 
+    @patch(HOOK_CLASS)
+    def test_xcom_sidecar_container_image_default(self, hook_mock):
+        hook_mock.return_value.get_xcom_sidecar_container_image.return_value = None
+        k = KubernetesPodOperator(
+            name="test",
+            task_id="task",
+            do_xcom_push=True,
+        )
+        pod = k.build_pod_request_obj(create_context(k))
+        assert pod.spec.containers[1].image == "alpine"
+
+    @patch(HOOK_CLASS)
+    def test_xcom_sidecar_container_image_custom(self, hook_mock):
+        hook_mock.return_value.get_xcom_sidecar_container_image.return_value = "private.repo/alpine:3.13"
+        k = KubernetesPodOperator(
+            name="test",
+            task_id="task",
+            do_xcom_push=True,
+        )
+        pod = k.build_pod_request_obj(create_context(k))
+        assert pod.spec.containers[1].image == "private.repo/alpine:3.13"
+
     def test_image_pull_policy_correctly_set(self):
         k = KubernetesPodOperator(
             task_id="task",