Repository: apache/airflow

Issue:
Move Kubernetes tests and kind setup out of the container to host
**Description**

We should move kind setup out of the container for CI to the host environment

**Use case / motivation**

Makes CI image smaller and setup easier for local testing.



Discussion:

Resolving patch:
diff --git a/airflow/kubernetes/refresh_config.py b/airflow/kubernetes/refresh_config.py
--- a/airflow/kubernetes/refresh_config.py
+++ b/airflow/kubernetes/refresh_config.py
@@ -25,6 +25,7 @@
 import os
 import time
 from datetime import datetime
+from typing import Optional
 
 import yaml
 from kubernetes.client import Configuration
@@ -91,7 +92,7 @@ def get_api_key_with_prefix(self, identifier):
         return Configuration.get_api_key_with_prefix(self, identifier)
 
 
-def _get_kube_config_loader_for_yaml_file(filename, **kwargs):
+def _get_kube_config_loader_for_yaml_file(filename, **kwargs) -> Optional[RefreshKubeConfigLoader]:
     """
     Adapted from the upstream _get_kube_config_loader_for_yaml_file function, changed
     KubeConfigLoader to RefreshKubeConfigLoader
@@ -115,5 +116,4 @@ def load_kube_config(client_configuration, config_file=None, context=None):
 
     loader = _get_kube_config_loader_for_yaml_file(
         config_file, active_context=context, config_persister=None)
-
     loader.load_and_set(client_configuration)
diff --git a/airflow/kubernetes/volume_mount.py b/airflow/kubernetes/volume_mount.py
--- a/airflow/kubernetes/volume_mount.py
+++ b/airflow/kubernetes/volume_mount.py
@@ -35,7 +35,7 @@ class VolumeMount(K8SModel):
     :param mount_path:
     :type mount_path: str
     :param sub_path: subpath within the volume mount
-    :type sub_path: str
+    :type sub_path: Optional[str]
     :param read_only: whether to access pod with read-only mode
     :type read_only: bool
     """


Test patch:
diff --git a/TESTING.rst b/TESTING.rst
--- a/TESTING.rst
+++ b/TESTING.rst
@@ -325,98 +325,224 @@ Those tests are skipped by default. You can enable them with ``--include-quarant
 can also decide to only run tests with ``-m quarantined`` flag to run only those tests.
 
 Running Tests with Kubernetes
------------------------------
+=============================
+
+Starting Kubernetes Cluster
+---------------------------
+
+For your testing you manage Kind cluster with ``kind-cluster`` breeze command:
+
+.. code-block:: bash
+
+    ./breeze kind-cluster [ start | stop | recreate | status ]
+
 
-Starting Kubernetes Cluster when Starting Breeze
-................................................
+The command allows you to start/stop/recreate/status Kind Kubernetes cluster
+in your docker engine as well as deploy airflow to use it for testing (with ``deploy`` command).
 
-To run Kubernetes in Breeze, you can start Breeze with the ``--kind-cluster-start`` switch. This
-automatically creates a Kind Kubernetes cluster in the same ``docker`` engine that is used to run Breeze.
-Setting up the Kubernetes cluster takes some time so the cluster continues running
-until it is stopped with the ``--kind-cluster-stop`` switch or until the ``--kind-cluster-recreate``
-switch is used rather than ``--kind-cluster-start``. Starting Breeze with the Kind Cluster automatically
-sets ``runtime`` to ``kubernetes`` (see below).
+Setting up the Kind Kubernetes cluster takes some time so once you started it, the cluster continues running
+until it is stopped with the ``kind-cluster stop`` command or until ``kind-cluster recreate``
+command is used (it will stop and recreate the cluster image).
 
-The cluster name follows the pattern ``airflow-python-X.Y.Z-vA.B.C`` where X.Y.Z is a Python version
+The cluster name follows the pattern ``airflow-python-X.Y-vA.B.C`` where X.Y is a Python version
 and A.B.C is a Kubernetes version. This way you can have multiple clusters set up and running at the same
 time for different Python versions and different Kubernetes versions.
 
-The Control Plane is available from inside the Docker image via ``<CLUSTER_NAME>-control-plane:6443``
-host:port, the worker of the Kind Cluster is available at  <CLUSTER_NAME>-worker
+The Control Plane is available from inside the Docker image via ``<KIND_CLUSTER_NAME>-control-plane:6443``
+host:port, the worker of the Kind Cluster is available at  <KIND_CLUSTER_NAME>-worker
 and the webserver port for the worker is 30809.
 
-After the Kubernetes Cluster is started, you need to deploy Airflow to the cluster:
+Deploying Airflow to Kubernetes Cluster
+---------------------------------------
 
-1. Build the image.
-2. Load it to the Kubernetes cluster.
-3. Deploy the Airflow application.
+Deploying Airflow to the Kubernetes cluster created is also done via ``kind-cluster`` breeze command:
 
-It can be done with a single script: ``./scripts/ci/in_container/deploy_airflow_to_kubernetes.sh``.
+.. code-block:: bash
 
-You can, however, work separately on the image in Kubernetes and deploying the Airflow app in the cluster.
+    ./breeze kind-cluster deploy
 
-Building and Loading Airflow Images to Kubernetes Cluster
-..............................................................
 
-Use the script ``./scripts/ci/in_container/kubernetes/docker/rebuild_airflow_image.sh`` that does the following:
+The deploy commands performs tthose steps:
 
-1. Rebuilds the latest ``apache/airflow:master-pythonX.Y-ci`` images using the latest sources.
-2. Builds a new Kubernetes image based on the  ``apache/airflow:master-pythonX.Y-ci`` using
+1. If needed, it rebuilds the latest ``apache/airflow:master-pythonX.Y`` production images using the
+   latest sources. You can also force the build with ``--force-build-image`` flag.
+2. Builds a new Kubernetes image based on the  ``apache/airflow:master-pythonX.Y`` using
    necessary scripts added to run in Kubernetes. The image is tagged as
-   ``apache/airflow:master-pythonX.Y-ci-kubernetes``.
+   ``apache/airflow:master-pythonX.Y-kubernetes``.
 3. Loads the image to the Kind Cluster using the ``kind load`` command.
-
-Deploying the Airflow Application in the Kubernetes Cluster
-...........................................................
-
-Use the script ``./scripts/ci/in_container/kubernetes/app/deploy_app.sh`` that does the following:
-
-1. Prepares Kubernetes resources by processing a template from the ``template`` directory and replacing
+4. Prepares Kubernetes resources by processing a template from the ``template`` directory and replacing
    variables with the right images and locations:
    - configmaps.yaml
    - airflow.yaml
-2. Uses the existing resources without replacing any variables inside:
+5. Uses the existing resources without replacing any variables inside:
    - secrets.yaml
    - postgres.yaml
    - volumes.yaml
-3. Applies all the resources to the Kind Cluster.
-4. Waits for all the applications to be ready and reachable.
+6. Applies all the resources to the Kind Cluster.
+7. Waits for all the applications to be ready and reachable.
 
-After the deployment is finished, you can run Kubernetes tests immediately in the same way as other tests.
-The Kubernetes tests are available in the ``tests/runtime/kubernetes`` folder.
 
-You can run all the integration tests for Kubernetes with ``pytest tests/runtime/kubernetes``.
+Running tests with Kubernetes Cluster
+-------------------------------------
 
+After the deployment is finished, you can run Kubernetes tests via ``scripts/ci/ci_run_kubernetes_tests.sh``.
 
-Running Runtime-Specific Tests
-------------------------------
+You can either run all tests or you can select which tests to run. You can also enter interactive virtualenv
+to run the tests manually one by one.
 
-Tests using a specific runtime are marked with a custom pytest marker ``pytest.mark.runtime``.
-The marker has a single parameter - the name of a runtime. At the moment the only supported runtime is
-``kubernetes``. This runtime is set when you run Breeze with one of the ``--kind-cluster-*`` flags.
-Runtime-specific tests run only when the selectd runtime is started.
 
+.. code-block:: bash
 
-.. code-block:: python
+    Running kubernetes tests
+
+      ./scripts/ci/ci_run_kubernetes_tests.sh                      - runs all kubernetes tests
+      ./scripts/ci/ci_run_kubernetes_tests.sh TEST [TEST ...]      - runs selected kubernetes tests (from kubernetes_tests folder)
+      ./scripts/ci/ci_run_kubernetes_tests.sh [-i|--interactive]   - Activates virtual environment ready to run tests and drops you in
+      ./scripts/ci/ci_run_kubernetes_tests.sh [--help]             - Prints this help message
+
+
+Typical testing pattern for Kubernetes tests
+--------------------------------------------
+
+The typical session for tests with Kubernetes looks like follows:
+
+1. Start the Kind cluster:
+
+.. code-block:: bash
+
+    ./breeze kind-cluster start
+
+    Starts Kind Kubernetes cluster
+
+       Use CI image.
+
+       Branch name:             master
+       Docker image:            apache/airflow:master-python3.7-ci
+
+       Airflow source version:  2.0.0.dev0
+       Python version:          3.7
+       DockerHub user:          apache
+       DockerHub repo:          airflow
+       Backend:                 postgres 9.6
+
+    No kind clusters found.
+
+    Creating cluster
+
+    Creating cluster "airflow-python-3.7-v1.17.0" ...
+     ✓ Ensuring node image (kindest/node:v1.17.0) 🖼
+     ✓ Preparing nodes 📦 📦
+     ✓ Writing configuration 📜
+     ✓ Starting control-plane 🕹️
+     ✓ Installing CNI 🔌
+    Could not read storage manifest, falling back on old k8s.io/host-path default ...
+     ✓ Installing StorageClass 💾
+     ✓ Joining worker nodes 🚜
+    Set kubectl context to "kind-airflow-python-3.7-v1.17.0"
+    You can now use your cluster with:
+
+    kubectl cluster-info --context kind-airflow-python-3.7-v1.17.0
+
+    Have a question, bug, or feature request? Let us know! https://kind.sigs.k8s.io/#community 🙂
+
+    Created cluster airflow-python-3.7-v1.17.0
+
+
+2. Check the status of the cluster
+
+.. code-block:: bash
+
+    ./breeze kind-cluster status
+
+    Checks status of Kind Kubernetes cluster
 
-    @pytest.mark.runtime("kubernetes")
-    class TestKubernetesExecutor(unittest.TestCase):
+       Use CI image.
 
+       Branch name:             master
+       Docker image:            apache/airflow:master-python3.7-ci
 
-You can use the custom ``--runtime`` switch in pytest to only run tests specific for that backend.
+       Airflow source version:  2.0.0.dev0
+       Python version:          3.7
+       DockerHub user:          apache
+       DockerHub repo:          airflow
+       Backend:                 postgres 9.6
+
+    airflow-python-3.7-v1.17.0-control-plane
+    airflow-python-3.7-v1.17.0-worker
+
+3. Deploy Airflow to the cluster
+
+.. code-block:: bash
+
+    ./breeze kind-cluster deploy
+
+4. Run Kubernetes tests
+
+Note that the tests are executed in production container not in the CI container.
+There is no need for the tests to run inside the Airflow CI container image as they only
+communicate with the Kubernetes-run Airflow deployed via the production image.
+Those Kubernetes tests require virtualenv to be created locally with airflow installed.
+The virtualenv required will be created automatically when the scripts are run.
+
+
+Either run all the tests:
+
+
+.. code-block:: bash
+
+    ./breeze kind-cluster test
+
+
+Or enter the interactive virtualenv (the environment is in ``.build/.kubernetes_venv`` folder:
 
-To run only kubernetes-runtime backend tests, enter:
 
 .. code-block:: bash
 
-    pytest --runtime kubernetes
+     ./scripts/ci/ci_run_kubernetes_tests.sh -i
+
+
+Once you enter the environment you get this information:
 
-**NOTE:** For convenience and faster search, all runtime tests are stored in the ``tests.runtime`` package. In this case, you
-can speed up the collection of tests by running:
 
 .. code-block:: bash
 
-    pytest --runtime kubernetes tests/runtime
+    Activating the virtual environment for kubernetes testing
+
+    You can run kubernetes testing via 'pytest kubernetes_tests/....'
+    You can add -s to see the output of your tests on screen
+
+    The webserver is available at http://localhost:30809/
+
+    User/password: airflow/airflow
+
+    You are entering the virtualenv now. Type exit to exit back to the original shell
+
+
+You can iterate with tests while you are in the virtualenv:
+
+
+.. code-block:: bash
+
+    pytest kubernetes_tests/test_kubernetes_executor.py::TestKubernetesExecutor::test_integration_run_dag_with_scheduler_failure -s
+
+
+You can modify the tests or KubernetesPodOperator and re-run them without re-deploying
+airflow to KinD cluster.
+
+However when you change the Airflow Kubernetes executor implementation you need to redeploy
+Airflow to the cluster.
+
+.. code-block:: bash
+
+    ./breeze kind-cluster deploy
+
+
+5. Stop KinD cluster when you are done
+
+.. code-block:: bash
+
+    ./breeze kind-cluster stop
+
 
 Airflow System Tests
 ====================
diff --git a/tests/runtime/__init__.py b/kubernetes_tests/__init__.py
similarity index 100%
rename from tests/runtime/__init__.py
rename to kubernetes_tests/__init__.py
diff --git a/tests/runtime/kubernetes/test_kubernetes_executor.py b/kubernetes_tests/test_kubernetes_executor.py
similarity index 76%
rename from tests/runtime/kubernetes/test_kubernetes_executor.py
rename to kubernetes_tests/test_kubernetes_executor.py
--- a/tests/runtime/kubernetes/test_kubernetes_executor.py
+++ b/kubernetes_tests/test_kubernetes_executor.py
@@ -20,16 +20,18 @@
 import unittest
 from subprocess import check_call, check_output
 
-import pytest
 import requests
 import requests.exceptions
 from requests.adapters import HTTPAdapter
 from urllib3.util.retry import Retry
 
-KUBERNETES_HOST = (os.environ.get('CLUSTER_NAME') or "docker") + "-worker:30809"
+KUBERNETES_HOST_PORT = (os.environ.get('CLUSTER_HOST') or "localhost") + ":30809"
+
+print()
+print(f"Cluster host/port used: ${KUBERNETES_HOST_PORT}")
+print()
 
 
-@pytest.mark.runtime("kubernetes")
 class TestKubernetesExecutor(unittest.TestCase):
 
     @staticmethod
@@ -56,7 +58,7 @@ def _get_session_with_retries(self):
 
     def _ensure_airflow_webserver_is_healthy(self):
         response = self.session.get(
-            "http://{host}/health".format(host=KUBERNETES_HOST),
+            "http://{host}/health".format(host=KUBERNETES_HOST_PORT),
             timeout=1,
         )
 
@@ -80,20 +82,18 @@ def monitor_task(self, host, execution_date, dag_id, task_id, expected_final_sta
 
             # Trigger a new dagrun
             try:
-                result = self.session.get(
-                    'http://{host}/api/experimental/dags/{dag_id}/'
-                    'dag_runs/{execution_date}/tasks/{task_id}'
-                    .format(host=host,
-                            dag_id=dag_id,
-                            execution_date=execution_date,
-                            task_id=task_id)
-                )
+                get_string = \
+                    f'http://{host}/api/experimental/dags/{dag_id}/' \
+                    f'dag_runs/{execution_date}/tasks/{task_id}'
+                print(f"Calling [monitor_task]#1 {get_string}")
+                result = self.session.get(get_string)
                 if result.status_code == 404:
                     check_call(["echo", "api returned 404."])
                     tries += 1
                     continue
                 self.assertEqual(result.status_code, 200, "Could not get the status")
                 result_json = result.json()
+                print(f"Received [monitor_task]#2: {result_json}")
                 state = result_json['state']
                 print("Attempt {}: Current state of operator is {}".format(tries, state))
 
@@ -116,18 +116,15 @@ def ensure_dag_expected_state(self, host, execution_date, dag_id,
         while tries < max_tries:
             time.sleep(5)
 
+            get_string = \
+                f'http://{host}/api/experimental/dags/{dag_id}/' \
+                f'dag_runs/{execution_date}'
+            print(f"Calling {get_string}")
             # Trigger a new dagrun
-            result = self.session.get(
-                'http://{host}/api/experimental/dags/{dag_id}/'
-                'dag_runs/{execution_date}'
-                .format(host=host,
-                        dag_id=dag_id,
-                        execution_date=execution_date)
-            )
-            print(result)
+            result = self.session.get(get_string)
             self.assertEqual(result.status_code, 200, "Could not get the status")
             result_json = result.json()
-            print(result_json)
+            print(f"Received: {result}")
             state = result_json['state']
             check_call(
                 ["echo", "Attempt {}: Current state of dag is {}".format(tries, state)])
@@ -142,53 +139,59 @@ def ensure_dag_expected_state(self, host, execution_date, dag_id,
         # Maybe check if we can retrieve the logs, but then we need to extend the API
 
     def start_dag(self, dag_id, host):
-        result = self.session.get(
-            'http://{host}/api/experimental/'
-            'dags/{dag_id}/paused/false'.format(host=host, dag_id=dag_id)
-        )
+        get_string = f'http://{host}/api/experimental/' \
+                     f'dags/{dag_id}/paused/false'
+        print(f"Calling [start_dag]#1 {get_string}")
+        result = self.session.get(get_string)
         try:
             result_json = result.json()
         except ValueError:
             result_json = str(result)
-
+        print(f"Received [start_dag]#1 {result_json}")
         self.assertEqual(result.status_code, 200, "Could not enable DAG: {result}"
                          .format(result=result_json))
-
+        post_string = f'http://{host}/api/experimental/' \
+                      f'dags/{dag_id}/dag_runs'
+        print(f"Calling [start_dag]#2 {post_string}")
         # Trigger a new dagrun
-        result = self.session.post(
-            'http://{host}/api/experimental/'
-            'dags/{dag_id}/dag_runs'.format(host=host, dag_id=dag_id),
-            json={}
-        )
+        result = self.session.post(post_string, json={})
         try:
             result_json = result.json()
         except ValueError:
             result_json = str(result)
-
+        print(f"Received [start_dag]#2 {result_json}")
         self.assertEqual(result.status_code, 200, "Could not trigger a DAG-run: {result}"
                          .format(result=result_json))
 
         time.sleep(1)
 
-        result = self.session.get(
-            'http://{}/api/experimental/latest_runs'.format(host)
-        )
+        get_string = f'http://{host}/api/experimental/latest_runs'
+        print(f"Calling [start_dag]#3 {get_string}")
+        result = self.session.get(get_string)
         self.assertEqual(result.status_code, 200, "Could not get the latest DAG-run:"
                                                   " {result}"
                          .format(result=result.json()))
         result_json = result.json()
+        print(f"Received: [start_dag]#3 {result_json}")
         return result_json
 
-    def test_integration_run_dag(self):
-        host = KUBERNETES_HOST
-        dag_id = 'example_kubernetes_executor_config'
-
+    def start_job_in_kubernetes(self, dag_id, host):
         result_json = self.start_dag(dag_id=dag_id, host=host)
-
         self.assertGreater(len(result_json['items']), 0)
+        execution_date = None
+        for dag_run in result_json['items']:
+            if dag_run['dag_id'] == dag_id:
+                execution_date = dag_run['execution_date']
+                break
+        self.assertIsNotNone(execution_date, f"No execution_date can be found for the dag with {dag_id}")
+        return execution_date
+
+    def test_integration_run_dag(self):
+        host = KUBERNETES_HOST_PORT
+        dag_id = 'example_kubernetes_executor_config'
 
-        execution_date = result_json['items'][0]['execution_date']
-        print("Found the job with execution date {}".format(execution_date))
+        execution_date = self.start_job_in_kubernetes(dag_id, host)
+        print(f"Found the job with execution date {execution_date}")
 
         # Wait some time for the operator to complete
         self.monitor_task(host=host,
@@ -203,15 +206,10 @@ def test_integration_run_dag(self):
                                        expected_final_state='success', timeout=100)
 
     def test_integration_run_dag_with_scheduler_failure(self):
-        host = KUBERNETES_HOST
+        host = KUBERNETES_HOST_PORT
         dag_id = 'example_kubernetes_executor_config'
 
-        result_json = self.start_dag(dag_id=dag_id, host=host)
-
-        self.assertGreater(len(result_json['items']), 0)
-
-        execution_date = result_json['items'][0]['execution_date']
-        print("Found the job with execution date {}".format(execution_date))
+        execution_date = self.start_job_in_kubernetes(dag_id, host)
 
         self._delete_airflow_pod()
 
diff --git a/tests/runtime/kubernetes/test_kubernetes_pod_operator.py b/kubernetes_tests/test_kubernetes_pod_operator.py
similarity index 85%
rename from tests/runtime/kubernetes/test_kubernetes_pod_operator.py
rename to kubernetes_tests/test_kubernetes_pod_operator.py
--- a/tests/runtime/kubernetes/test_kubernetes_pod_operator.py
+++ b/kubernetes_tests/test_kubernetes_pod_operator.py
@@ -1,3 +1,4 @@
+# pylint: disable=unused-argument
 # Licensed to the Apache Software Foundation (ASF) under one
 # or more contributor license agreements.  See the NOTICE file
 # distributed with this work for additional information
@@ -24,7 +25,6 @@
 
 import kubernetes.client.models as k8s
 import pendulum
-import pytest
 from kubernetes.client.api_client import ApiClient
 from kubernetes.client.rest import ApiException
 
@@ -42,8 +42,23 @@
 from airflow.version import version as airflow_version
 
 
-@pytest.mark.runtime("kubernetes")
-class TestKubernetesPodOperator(unittest.TestCase):
+# noinspection DuplicatedCode
+def create_context(task):
+    dag = DAG(dag_id="dag")
+    tzinfo = pendulum.timezone("Europe/Amsterdam")
+    execution_date = timezone.datetime(2016, 1, 1, 1, 0, 0, tzinfo=tzinfo)
+    task_instance = TaskInstance(task=task,
+                                 execution_date=execution_date)
+    return {
+        "dag": dag,
+        "ts": execution_date.isoformat(),
+        "task": task,
+        "ti": task_instance,
+    }
+
+
+# noinspection DuplicatedCode,PyUnusedLocal
+class TestKubernetesPodOperatorSystem(unittest.TestCase):
 
     def setUp(self):
         self.maxDiff = None  # pylint: disable=invalid-name
@@ -92,19 +107,6 @@ def tearDown(self) -> None:
         client = kube_client.get_kube_client(in_cluster=False)
         client.delete_collection_namespaced_pod(namespace="default")
 
-    def create_context(self, task):
-        dag = DAG(dag_id="dag")
-        tzinfo = pendulum.timezone("Europe/Amsterdam")
-        execution_date = timezone.datetime(2016, 1, 1, 1, 0, 0, tzinfo=tzinfo)
-        task_instance = TaskInstance(task=task,
-                                     execution_date=execution_date)
-        return {
-            "dag": dag,
-            "ts": execution_date.isoformat(),
-            "task": task,
-            "ti": task_instance,
-        }
-
     def test_do_xcom_push_defaults_false(self):
         new_config_path = '/tmp/kube_config'
         old_config_path = os.path.expanduser('~/.kube/config')
@@ -141,98 +143,11 @@ def test_config_path_move(self):
             do_xcom_push=False,
             config_file=new_config_path,
         )
-        context = self.create_context(k)
+        context = create_context(k)
         k.execute(context)
         actual_pod = self.api_client.sanitize_for_serialization(k.pod)
         self.assertEqual(self.expected_pod, actual_pod)
 
-    @mock.patch("airflow.kubernetes.pod_launcher.PodLauncher.start_pod")
-    @mock.patch("airflow.kubernetes.pod_launcher.PodLauncher.monitor_pod")
-    @mock.patch("airflow.kubernetes.kube_client.get_kube_client")
-    def test_config_path(self, client_mock, monitor_mock, start_mock):  # pylint: disable=unused-argument
-        from airflow.utils.state import State
-
-        file_path = "/tmp/fake_file"
-        k = KubernetesPodOperator(
-            namespace='default',
-            image="ubuntu:16.04",
-            cmds=["bash", "-cx"],
-            arguments=["echo 10"],
-            labels={"foo": "bar"},
-            name="test",
-            task_id="task",
-            in_cluster=False,
-            do_xcom_push=False,
-            config_file=file_path,
-            cluster_context='default',
-        )
-        monitor_mock.return_value = (State.SUCCESS, None)
-        client_mock.list_namespaced_pod.return_value = []
-        context = self.create_context(k)
-        k.execute(context=context)
-        client_mock.assert_called_once_with(
-            in_cluster=False,
-            cluster_context='default',
-            config_file=file_path,
-        )
-
-    @mock.patch("airflow.kubernetes.pod_launcher.PodLauncher.start_pod")
-    @mock.patch("airflow.kubernetes.pod_launcher.PodLauncher.monitor_pod")
-    @mock.patch("airflow.kubernetes.kube_client.get_kube_client")
-    def test_image_pull_secrets_correctly_set(self, mock_client, monitor_mock, start_mock):
-        from airflow.utils.state import State
-
-        fake_pull_secrets = "fakeSecret"
-        k = KubernetesPodOperator(
-            namespace='default',
-            image="ubuntu:16.04",
-            cmds=["bash", "-cx"],
-            arguments=["echo 10"],
-            labels={"foo": "bar"},
-            name="test",
-            task_id="task",
-            in_cluster=False,
-            do_xcom_push=False,
-            image_pull_secrets=fake_pull_secrets,
-            cluster_context='default',
-        )
-        monitor_mock.return_value = (State.SUCCESS, None)
-        context = self.create_context(k)
-        k.execute(context=context)
-        self.assertEqual(
-            start_mock.call_args[0][0].spec.image_pull_secrets,
-            [k8s.V1LocalObjectReference(name=fake_pull_secrets)]
-        )
-
-    @mock.patch("airflow.kubernetes.pod_launcher.PodLauncher.start_pod")
-    @mock.patch("airflow.kubernetes.pod_launcher.PodLauncher.monitor_pod")
-    @mock.patch("airflow.kubernetes.pod_launcher.PodLauncher.delete_pod")
-    @mock.patch("airflow.kubernetes.kube_client.get_kube_client")
-    def test_pod_delete_even_on_launcher_error(
-            self,
-            mock_client,
-            delete_pod_mock,
-            monitor_pod_mock,
-            start_pod_mock):  # pylint: disable=unused-argument
-        k = KubernetesPodOperator(
-            namespace='default',
-            image="ubuntu:16.04",
-            cmds=["bash", "-cx"],
-            arguments=["echo 10"],
-            labels={"foo": "bar"},
-            name="test",
-            task_id="task",
-            in_cluster=False,
-            do_xcom_push=False,
-            cluster_context='default',
-            is_delete_operator_pod=True,
-        )
-        monitor_pod_mock.side_effect = AirflowException('fake failure')
-        with self.assertRaises(AirflowException):
-            context = self.create_context(k)
-            k.execute(context=context)
-        assert delete_pod_mock.called
-
     def test_working_pod(self):
         k = KubernetesPodOperator(
             namespace='default',
@@ -245,7 +160,7 @@ def test_working_pod(self):
             in_cluster=False,
             do_xcom_push=False,
         )
-        context = self.create_context(k)
+        context = create_context(k)
         k.execute(context)
         actual_pod = self.api_client.sanitize_for_serialization(k.pod)
         self.assertEqual(self.expected_pod['spec'], actual_pod['spec'])
@@ -264,7 +179,7 @@ def test_delete_operator_pod(self):
             do_xcom_push=False,
             is_delete_operator_pod=True,
         )
-        context = self.create_context(k)
+        context = create_context(k)
         k.execute(context)
         actual_pod = self.api_client.sanitize_for_serialization(k.pod)
         self.assertEqual(self.expected_pod['spec'], actual_pod['spec'])
@@ -283,7 +198,7 @@ def test_pod_hostnetwork(self):
             do_xcom_push=False,
             hostnetwork=True,
         )
-        context = self.create_context(k)
+        context = create_context(k)
         k.execute(context)
         actual_pod = self.api_client.sanitize_for_serialization(k.pod)
         self.expected_pod['spec']['hostNetwork'] = True
@@ -305,7 +220,7 @@ def test_pod_dnspolicy(self):
             hostnetwork=True,
             dnspolicy=dns_policy
         )
-        context = self.create_context(k)
+        context = create_context(k)
         k.execute(context)
         actual_pod = self.api_client.sanitize_for_serialization(k.pod)
         self.expected_pod['spec']['hostNetwork'] = True
@@ -327,7 +242,7 @@ def test_pod_schedulername(self):
             do_xcom_push=False,
             schedulername=scheduler_name
         )
-        context = self.create_context(k)
+        context = create_context(k)
         k.execute(context)
         actual_pod = self.api_client.sanitize_for_serialization(k.pod)
         self.expected_pod['spec']['schedulerName'] = scheduler_name
@@ -349,7 +264,7 @@ def test_pod_node_selectors(self):
             do_xcom_push=False,
             node_selectors=node_selectors,
         )
-        context = self.create_context(k)
+        context = create_context(k)
         k.execute(context)
         actual_pod = self.api_client.sanitize_for_serialization(k.pod)
         self.expected_pod['spec']['nodeSelector'] = node_selectors
@@ -376,7 +291,7 @@ def test_pod_resources(self):
             do_xcom_push=False,
             resources=resources,
         )
-        context = self.create_context(k)
+        context = create_context(k)
         k.execute(context)
         actual_pod = self.api_client.sanitize_for_serialization(k.pod)
         self.expected_pod['spec']['containers'][0]['resources'] = {
@@ -424,7 +339,7 @@ def test_pod_affinity(self):
             do_xcom_push=False,
             affinity=affinity,
         )
-        context = self.create_context(k)
+        context = create_context(k)
         k.execute(context=context)
         actual_pod = self.api_client.sanitize_for_serialization(k.pod)
         self.expected_pod['spec']['affinity'] = affinity
@@ -445,7 +360,7 @@ def test_port(self):
             do_xcom_push=False,
             ports=[port],
         )
-        context = self.create_context(k)
+        context = create_context(k)
         k.execute(context=context)
         actual_pod = self.api_client.sanitize_for_serialization(k.pod)
         self.expected_pod['spec']['containers'][0]['ports'] = [{
@@ -483,7 +398,7 @@ def test_volume_mount(self):
                 in_cluster=False,
                 do_xcom_push=False,
             )
-            context = self.create_context(k)
+            context = create_context(k)
             k.execute(context=context)
             mock_logger.info.assert_any_call(b"retrieved from mount\n")
             actual_pod = self.api_client.sanitize_for_serialization(k.pod)
@@ -519,7 +434,7 @@ def test_run_as_user_root(self):
             do_xcom_push=False,
             security_context=security_context,
         )
-        context = self.create_context(k)
+        context = create_context(k)
         k.execute(context)
         actual_pod = self.api_client.sanitize_for_serialization(k.pod)
         self.expected_pod['spec']['securityContext'] = security_context
@@ -544,7 +459,7 @@ def test_run_as_user_non_root(self):
             do_xcom_push=False,
             security_context=security_context,
         )
-        context = self.create_context(k)
+        context = create_context(k)
         k.execute(context)
         actual_pod = self.api_client.sanitize_for_serialization(k.pod)
         self.expected_pod['spec']['securityContext'] = security_context
@@ -569,7 +484,7 @@ def test_fs_group(self):
             do_xcom_push=False,
             security_context=security_context,
         )
-        context = self.create_context(k)
+        context = create_context(k)
         k.execute(context)
         actual_pod = self.api_client.sanitize_for_serialization(k.pod)
         self.expected_pod['spec']['securityContext'] = security_context
@@ -590,7 +505,7 @@ def test_faulty_image(self):
             startup_timeout_seconds=5,
         )
         with self.assertRaises(AirflowException):
-            context = self.create_context(k)
+            context = create_context(k)
             k.execute(context)
             actual_pod = self.api_client.sanitize_for_serialization(k.pod)
             self.expected_pod['spec']['containers'][0]['image'] = bad_image_name
@@ -612,7 +527,7 @@ def test_faulty_service_account(self):
             service_account_name=bad_service_account_name,
         )
         with self.assertRaises(ApiException):
-            context = self.create_context(k)
+            context = create_context(k)
             k.execute(context)
             actual_pod = self.api_client.sanitize_for_serialization(k.pod)
             self.expected_pod['spec']['serviceAccountName'] = bad_service_account_name
@@ -635,7 +550,7 @@ def test_pod_failure(self):
             do_xcom_push=False,
         )
         with self.assertRaises(AirflowException):
-            context = self.create_context(k)
+            context = create_context(k)
             k.execute(context)
             actual_pod = self.api_client.sanitize_for_serialization(k.pod)
             self.expected_pod['spec']['containers'][0]['args'] = bad_internal_command
@@ -655,14 +570,14 @@ def test_xcom_push(self):
             in_cluster=False,
             do_xcom_push=True,
         )
-        context = self.create_context(k)
+        context = create_context(k)
         self.assertEqual(k.execute(context), json.loads(return_value))
         actual_pod = self.api_client.sanitize_for_serialization(k.pod)
         volume = self.api_client.sanitize_for_serialization(PodDefaults.VOLUME)
         volume_mount = self.api_client.sanitize_for_serialization(PodDefaults.VOLUME_MOUNT)
         container = self.api_client.sanitize_for_serialization(PodDefaults.SIDECAR_CONTAINER)
         self.expected_pod['spec']['containers'][0]['args'] = args
-        self.expected_pod['spec']['containers'][0]['volumeMounts'].insert(0, volume_mount)
+        self.expected_pod['spec']['containers'][0]['volumeMounts'].insert(0, volume_mount)  # noqa
         self.expected_pod['spec']['volumes'].insert(0, volume)
         self.expected_pod['spec']['containers'].append(container)
         self.assertEqual(self.expected_pod, actual_pod)
@@ -690,7 +605,7 @@ def test_envs_from_configmaps(self, mock_client, mock_monitor, mock_start):
         )
         # THEN
         mock_monitor.return_value = (State.SUCCESS, None)
-        context = self.create_context(k)
+        context = create_context(k)
         k.execute(context)
         self.assertEqual(
             mock_start.call_args[0][0].spec.containers[0].env_from,
@@ -722,7 +637,7 @@ def test_envs_from_secrets(self, mock_client, monitor_mock, start_mock):
         )
         # THEN
         monitor_mock.return_value = (State.SUCCESS, None)
-        context = self.create_context(k)
+        context = create_context(k)
         k.execute(context)
         self.assertEqual(
             start_mock.call_args[0][0].spec.containers[0].env_from,
@@ -797,7 +712,7 @@ def test_init_container(self):
             in_cluster=False,
             do_xcom_push=False,
         )
-        context = self.create_context(k)
+        context = create_context(k)
         k.execute(context)
         actual_pod = self.api_client.sanitize_for_serialization(k.pod)
         self.expected_pod['spec']['initContainers'] = [expected_init_container]
@@ -824,7 +739,7 @@ def test_pod_template_file(
             do_xcom_push=True
         )
         monitor_mock.return_value = (State.SUCCESS, None)
-        context = self.create_context(k)
+        context = create_context(k)
         k.execute(context)
         actual_pod = self.api_client.sanitize_for_serialization(k.pod)
         self.assertEqual({
@@ -889,7 +804,7 @@ def test_pod_priority_class_name(
         )
 
         monitor_mock.return_value = (State.SUCCESS, None)
-        context = self.create_context(k)
+        context = create_context(k)
         k.execute(context)
         actual_pod = self.api_client.sanitize_for_serialization(k.pod)
         self.expected_pod['spec']['priorityClassName'] = priority_class_name
diff --git a/scripts/ci/ci_run_airflow_testing.sh b/scripts/ci/ci_run_airflow_testing.sh
--- a/scripts/ci/ci_run_airflow_testing.sh
+++ b/scripts/ci/ci_run_airflow_testing.sh
@@ -18,7 +18,7 @@
 export VERBOSE=${VERBOSE:="false"}
 
 function run_airflow_testing_in_docker_with_kubernetes() {
-    export KUBERNETES_MODE=${KUBERNETES_MODE:="git_mode"}
+    export KUBERNETES_MODE=${KUBERNETES_MODE:="image"}
     export KUBERNETES_VERSION=${KUBERNETES_VERSION:="v1.15.3"}
 
     # shellcheck disable=SC2016
@@ -149,3 +149,34 @@ do
     exit ${EXIT_CODE}
 done
 set -u
+
+#TODO FIXME
+if [[ "${INTEGRATIONS[*]}" == *"kubernetes"* ]]; then
+    export KUBERNETES_MODE=${KUBERNETES_MODE:="image"}
+    export KUBERNETES_VERSION=${KUBERNETES_VERSION:="v1.15.3"}
+    export KIND_CLUSTER_NAME="airflow-python-${PYTHON_MAJOR_MINOR_VERSION}-${KUBERNETES_VERSION}"
+
+    if [[ ${ENABLE_KIND_CLUSTER} == "true" ]]; then
+        install_kind_in_host
+        "${MY_DIR}/setup_kind_cluster_in_host.sh"
+    fi
+
+    # adding trap to exiting trap
+    HANDLERS="$( trap -p EXIT | cut -f2 -d \' )"
+    # shellcheck disable=SC2064
+    trap "${HANDLERS}${HANDLERS:+;}send_kubernetes_logs_to_file_io" EXIT
+fi
+
+set +u
+# shellcheck disable=SC2016
+docker-compose --log-level INFO \
+  -f "${MY_DIR}/docker-compose/base.yml" \
+  -f "${MY_DIR}/docker-compose/backend-${BACKEND}.yml" \
+  "${INTEGRATIONS[@]}" \
+  "${DOCKER_COMPOSE_LOCAL[@]}" \
+     run airflow \
+       '/opt/airflow/scripts/ci/in_container/entrypoint_ci.sh "${@}"' \
+       /opt/airflow/scripts/ci/in_container/entrypoint_ci.sh "${@}"
+     # Note the command is there twice (!) because it is passed via bash -c
+     # and bash -c starts passing parameters from $0. TODO: fixme
+set -u
diff --git a/scripts/ci/ci_run_kubernetes_tests.sh b/scripts/ci/ci_run_kubernetes_tests.sh
new file mode 100755
--- /dev/null
+++ b/scripts/ci/ci_run_kubernetes_tests.sh
@@ -0,0 +1,105 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+# shellcheck source=scripts/ci/_script_init.sh
+. "$( dirname "${BASH_SOURCE[0]}" )/_script_init.sh"
+
+INTERACTIVE="false"
+
+declare -a TESTS
+declare -a PYTEST_ARGS
+
+TESTS=()
+
+if [[ $# != 0 ]]; then
+    if [[ $1 == "--help" || $1 == "-h" ]]; then
+        echo
+        echo "Running kubernetes tests"
+        echo
+        echo "    $0                      - runs all kubernetes tests"
+        echo "    $0 TEST [TEST ...]      - runs selected kubernetes tests (from kubernetes_tests folder)"
+        echo "    $0 [-i|--interactive]   - Activates virtual environment ready to run tests and drops you in"
+        echo "    $0 [--help]             - Prints this help message"
+        echo
+        exit
+    elif [[ $1 == "--interactive" || $1 == "-i" ]]; then
+        echo
+        echo "Entering interactive environment for kubernetes testing"
+        echo
+        INTERACTIVE="true"
+    else
+        TESTS=("${@}")
+    fi
+    PYTEST_ARGS=(
+        "--pythonwarnings=ignore::DeprecationWarning"
+        "--pythonwarnings=ignore::PendingDeprecationWarning"
+    )
+else
+    TESTS=("kubernetes_tests")
+    PYTEST_ARGS=(
+        "--verbosity=1"
+        "--strict-markers"
+        "--durations=100"
+        "--cov=airflow/"
+        "--cov-config=.coveragerc"
+        "--cov-report=html:airflow/www/static/coverage/"
+        "--color=yes"
+        "--maxfail=50"
+        "--pythonwarnings=ignore::DeprecationWarning"
+        "--pythonwarnings=ignore::PendingDeprecationWarning"
+        )
+
+fi
+
+get_ci_environment
+
+cd "${AIRFLOW_SOURCES}" || exit 1
+
+VIRTUALENV_PATH="${BUILD_CACHE_DIR}/.kubernetes_venv"
+
+if [[ ! -d ${VIRTUALENV_PATH} ]]; then
+    echo
+    echo "Creating virtualenv at ${VIRTUALENV_PATH}"
+    echo
+    python -m venv "${VIRTUALENV_PATH}"
+fi
+
+. "${VIRTUALENV_PATH}/bin/activate"
+
+pip install pytest freezegun pytest-cov \
+    --constraint "requirements/requirements-python${PYTHON_MAJOR_MINOR_VERSION}.txt"
+
+pip install -e ".[kubernetes]" \
+    --constraint "requirements/requirements-python${PYTHON_MAJOR_MINOR_VERSION}.txt"
+
+if [[ ${INTERACTIVE} == "true" ]]; then
+    echo
+    echo "Activating the virtual environment for kubernetes testing"
+    echo
+    echo "You can run kubernetes testing via 'pytest kubernetes_tests/....'"
+    echo "You can add -s to see the output of your tests on screen"
+    echo
+    echo "The webserver is available at http://localhost:30809/"
+    echo
+    echo "User/password: airflow/airflow"
+    echo
+    echo "You are entering the virtualenv now. Type exit to exit back to the original shell"
+    echo
+    exec "${SHELL}"
+else
+    pytest "${PYTEST_ARGS[@]}" "${TESTS[@]}"
+fi
diff --git a/scripts/ci/in_container/run_ci_tests.sh b/scripts/ci/in_container/run_ci_tests.sh
--- a/scripts/ci/in_container/run_ci_tests.sh
+++ b/scripts/ci/in_container/run_ci_tests.sh
@@ -41,8 +41,4 @@ if [[ ${CI} == "true" ]]; then
     send_airflow_logs_to_file_io
 fi
 
-if [[ ${CI} == "true" && ${ENABLE_KIND_CLUSTER} == "true" ]]; then
-    send_kubernetes_logs_to_file_io
-fi
-
 exit "${RES}"
diff --git a/tests/conftest.py b/tests/conftest.py
--- a/tests/conftest.py
+++ b/tests/conftest.py
@@ -143,12 +143,6 @@ def pytest_addoption(parser):
         metavar="BACKEND",
         help="only run tests matching the backend: [sqlite,postgres,mysql].",
     )
-    group.addoption(
-        "--runtime",
-        action="store",
-        metavar="RUNTIME",
-        help="only run tests matching the runtime: [kubernetes].",
-    )
     group.addoption(
         "--system",
         action="append",
@@ -247,9 +241,6 @@ def pytest_configure(config):
     config.addinivalue_line(
         "markers", "backend(name): mark test to run with named backend"
     )
-    config.addinivalue_line(
-        "markers", "runtime(name): mark test to run with named runtime"
-    )
     config.addinivalue_line(
         "markers", "system(name): mark test to run with named system"
     )
@@ -289,16 +280,6 @@ def skip_if_not_marked_with_backend(selected_backend, item):
                 format(backend=selected_backend, item=item))
 
 
-def skip_if_not_marked_with_runtime(selected_runtime, item):
-    for marker in item.iter_markers(name="runtime"):
-        runtime_name = marker.args[0]
-        if runtime_name == selected_runtime:
-            return
-    pytest.skip("The test is skipped because it has not been selected via --runtime switch. "
-                "Only tests marked with pytest.mark.runtime('{runtime}') are run: {item}".
-                format(runtime=selected_runtime, item=item))
-
-
 def skip_if_not_marked_with_system(selected_systems, item):
     for marker in item.iter_markers(name="system"):
         systems_name = marker.args[0]
@@ -345,19 +326,6 @@ def skip_if_integration_disabled(marker, item):
                            integration_name=integration_name, item=item))
 
 
-def skip_if_runtime_disabled(marker, item):
-    runtime_name = marker.args[0]
-    environment_variable_name = "RUNTIME"
-    environment_variable_value = os.environ.get(environment_variable_name)
-    if not environment_variable_value or environment_variable_value != runtime_name:
-        pytest.skip("The test requires {runtime_name} integration started and "
-                    "{name} environment variable to be set to true (it is '{value}')."
-                    " It can be set by specifying '--kind-cluster-start' at breeze startup"
-                    ": {item}".
-                    format(name=environment_variable_name, value=environment_variable_value,
-                           runtime_name=runtime_name, item=item))
-
-
 def skip_if_wrong_backend(marker, item):
     valid_backend_names = marker.args
     environment_variable_name = "BACKEND"
@@ -406,11 +374,6 @@ def pytest_runtest_setup(item):
     selected_backend = item.config.getoption("--backend")
     if selected_backend:
         skip_if_not_marked_with_backend(selected_backend, item)
-    for marker in item.iter_markers(name="runtime"):
-        skip_if_runtime_disabled(marker, item)
-    selected_runtime = item.config.getoption("--runtime")
-    if selected_runtime:
-        skip_if_not_marked_with_runtime(selected_runtime, item)
     if not include_long_running:
         skip_long_running_test(item)
     if not include_quarantined:
diff --git a/tests/providers/cncf/kubernetes/operators/test_kubernetes_pod.py b/tests/providers/cncf/kubernetes/operators/test_kubernetes_pod.py
new file mode 100644
--- /dev/null
+++ b/tests/providers/cncf/kubernetes/operators/test_kubernetes_pod.py
@@ -0,0 +1,134 @@
+# pylint: disable=unused-argument
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import unittest
+from unittest import mock
+
+import kubernetes.client.models as k8s
+import pendulum
+
+from airflow.exceptions import AirflowException
+from airflow.models import DAG, TaskInstance
+from airflow.providers.cncf.kubernetes.operators.kubernetes_pod import KubernetesPodOperator
+from airflow.utils import timezone
+
+
+# noinspection PyUnusedLocal
+class TestKubernetesPodOperator(unittest.TestCase):
+
+    # noinspection DuplicatedCode
+    @staticmethod
+    def create_context(task):
+        dag = DAG(dag_id="dag")
+        tzinfo = pendulum.timezone("Europe/Amsterdam")
+        execution_date = timezone.datetime(2016, 1, 1, 1, 0, 0, tzinfo=tzinfo)
+        task_instance = TaskInstance(task=task,
+                                     execution_date=execution_date)
+        return {
+            "dag": dag,
+            "ts": execution_date.isoformat(),
+            "task": task,
+            "ti": task_instance,
+        }
+
+    @mock.patch("airflow.kubernetes.pod_launcher.PodLauncher.start_pod")
+    @mock.patch("airflow.kubernetes.pod_launcher.PodLauncher.monitor_pod")
+    @mock.patch("airflow.kubernetes.kube_client.get_kube_client")
+    def test_config_path(self, client_mock, monitor_mock, start_mock):  # pylint: disable=unused-argument
+        from airflow.utils.state import State
+
+        file_path = "/tmp/fake_file"
+        k = KubernetesPodOperator(
+            namespace='default',
+            image="ubuntu:16.04",
+            cmds=["bash", "-cx"],
+            arguments=["echo 10"],
+            labels={"foo": "bar"},
+            name="test",
+            task_id="task",
+            in_cluster=False,
+            do_xcom_push=False,
+            config_file=file_path,
+            cluster_context='default',
+        )
+        monitor_mock.return_value = (State.SUCCESS, None)
+        client_mock.list_namespaced_pod.return_value = []
+        context = self.create_context(k)
+        k.execute(context=context)
+        client_mock.assert_called_once_with(
+            in_cluster=False,
+            cluster_context='default',
+            config_file=file_path,
+        )
+
+    @mock.patch("airflow.kubernetes.pod_launcher.PodLauncher.start_pod")
+    @mock.patch("airflow.kubernetes.pod_launcher.PodLauncher.monitor_pod")
+    @mock.patch("airflow.kubernetes.kube_client.get_kube_client")
+    def test_image_pull_secrets_correctly_set(self, mock_client, monitor_mock, start_mock):
+        from airflow.utils.state import State
+
+        fake_pull_secrets = "fakeSecret"
+        k = KubernetesPodOperator(
+            namespace='default',
+            image="ubuntu:16.04",
+            cmds=["bash", "-cx"],
+            arguments=["echo 10"],
+            labels={"foo": "bar"},
+            name="test",
+            task_id="task",
+            in_cluster=False,
+            do_xcom_push=False,
+            image_pull_secrets=fake_pull_secrets,
+            cluster_context='default',
+        )
+        monitor_mock.return_value = (State.SUCCESS, None)
+        context = self.create_context(k)
+        k.execute(context=context)
+        self.assertEqual(
+            start_mock.call_args[0][0].spec.image_pull_secrets,
+            [k8s.V1LocalObjectReference(name=fake_pull_secrets)]
+        )
+
+    @mock.patch("airflow.kubernetes.pod_launcher.PodLauncher.start_pod")
+    @mock.patch("airflow.kubernetes.pod_launcher.PodLauncher.monitor_pod")
+    @mock.patch("airflow.kubernetes.pod_launcher.PodLauncher.delete_pod")
+    @mock.patch("airflow.kubernetes.kube_client.get_kube_client")
+    def test_pod_delete_even_on_launcher_error(
+            self,
+            mock_client,
+            delete_pod_mock,
+            monitor_pod_mock,
+            start_pod_mock):
+        k = KubernetesPodOperator(
+            namespace='default',
+            image="ubuntu:16.04",
+            cmds=["bash", "-cx"],
+            arguments=["echo 10"],
+            labels={"foo": "bar"},
+            name="test",
+            task_id="task",
+            in_cluster=False,
+            do_xcom_push=False,
+            cluster_context='default',
+            is_delete_operator_pod=True,
+        )
+        monitor_pod_mock.side_effect = AirflowException('fake failure')
+        with self.assertRaises(AirflowException):
+            context = self.create_context(k)
+            k.execute(context=context)
+        assert delete_pod_mock.called
diff --git a/tests/providers/cncf/kubernetes/operators/test_spark_kubernetes_system.py b/tests/providers/cncf/kubernetes/operators/test_spark_kubernetes_system.py
--- a/tests/providers/cncf/kubernetes/operators/test_spark_kubernetes_system.py
+++ b/tests/providers/cncf/kubernetes/operators/test_spark_kubernetes_system.py
@@ -1,4 +1,3 @@
-#
 # Licensed to the Apache Software Foundation (ASF) under one
 # or more contributor license agreements.  See the NOTICE file
 # distributed with this work for additional information
@@ -55,7 +54,6 @@ def kubectl_delete_list(manifests):
         subprocess.run(command, check=True)
 
 
-@pytest.mark.runtime("kubernetes")
 @pytest.mark.system("cncf.kubernetes")
 class SparkKubernetesExampleDagsSystemTest(SystemTest):
 
diff --git a/tests/runtime/kubernetes/__init__.py b/tests/runtime/kubernetes/__init__.py
deleted file mode 100644
--- a/tests/runtime/kubernetes/__init__.py
+++ /dev/null
@@ -1,17 +0,0 @@
-#
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements.  See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership.  The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License.  You may obtain a copy of the License at
-#
-#   http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied.  See the License for the
-# specific language governing permissions and limitations
-# under the License.
diff --git a/tests/test_project_structure.py b/tests/test_project_structure.py
--- a/tests/test_project_structure.py
+++ b/tests/test_project_structure.py
@@ -33,7 +33,6 @@
     'tests/providers/apache/hive/operators/test_vertica_to_hive.py',
     'tests/providers/apache/pig/operators/test_pig.py',
     'tests/providers/apache/spark/hooks/test_spark_jdbc_script.py',
-    'tests/providers/cncf/kubernetes/operators/test_kubernetes_pod.py',
     'tests/providers/google/cloud/operators/test_datastore.py',
     'tests/providers/google/cloud/operators/test_sql_to_gcs.py',
     'tests/providers/google/cloud/utils/test_field_sanitizer.py',