Repository: zenml-io/zenml

Issue:
[BUG]: K8s orchestrator for scheduled pipeline: AttributeError: 'BatchV1Api' object has no attribute 'create_namespaced_cron_job'
### Contact Details [Optional]

francois.serra@adeo.com

### System Information

ZENML_LOCAL_VERSION: 0.43.0
ZENML_SERVER_VERSION: 0.43.0
PYTHON_VERSION: 3.10.8
ENVIRONMENT: native
SYSTEM_INFO: {'os': 'linux', 'linux_distro': 'ubuntu', 'linux_distro_like': 'debian', 'linux_distro_version': '22.04'}
INTEGRATIONS: ['kaniko', 'kubernetes', 'pillow', 'scipy', 'seldon']
PACKAGES: {'babel': '2.12.1', 'gitpython': '3.1.33', 'jinja2': '3.1.2', 'lunarcalendar': '0.0.9', 'mako': '1.2.4', 'markupsafe': '2.1.3', 'pillow': '10.0.0', 'pymeeus': '0.5.12', 
'pymysql': '1.0.3', 'pyyaml': '6.0.1', 'pygments': '2.16.1', 'sqlalchemy': '1.4.41', 'sqlalchemy-utils': '0.38.3', 'send2trash': '1.8.2', 'unidecode': '1.3.6', 'aiohttp': '3.8.5', 
'aiosignal': '1.3.1', 'alembic': '1.8.1', 'analytics-python': '1.4.post1', 'anyio': '4.0.0', 'argon2-cffi': '23.1.0', 'argon2-cffi-bindings': '21.2.0', 'arrow': '1.2.3', 'asttokens': 
'2.2.1', 'async-lru': '2.0.4', 'async-timeout': '4.0.3', 'attrs': '23.1.0', 'azure-common': '1.1.28', 'azure-core': '1.29.3', 'azure-mgmt-core': '1.4.0', 'azure-mgmt-resource': 
'23.0.1', 'backcall': '0.2.0', 'backoff': '1.10.0', 'bcrypt': '4.0.1', 'beautifulsoup4': '4.12.2', 'bleach': '6.0.0', 'cachetools': '5.3.1', 'certifi': '2023.7.22', 'cffi': '1.15.1', 
'charset-normalizer': '3.2.0', 'click': '8.1.3', 'click-params': '0.3.0', 'cloudpickle': '2.2.1', 'cmaes': '0.10.0', 'cmdstanpy': '1.1.0', 'colorlog': '6.7.0', 'comm': '0.1.4', 
'commonmark': '0.9.1', 'contourpy': '1.1.0', 'convertdate': '2.4.0', 'coverage': '7.3.0', 'cycler': '0.11.0', 'db-dtypes': '1.1.1', 'debugpy': '1.6.7.post1', 'decorator': '5.1.1', 
'defusedxml': '0.7.1', 'distro': '1.8.0', 'docker': '6.1.3', 'dtaidistance': '2.3.10', 'easyplan-ml': '0.1.0', 'ephem': '4.1.4', 'exceptiongroup': '1.1.3', 'executing': '1.2.0', 
'fastjsonschema': '2.18.0', 'fonttools': '4.42.1', 'fqdn': '1.5.1', 'frozenlist': '1.4.0', 'fsspec': '2023.6.0', 'gcsfs': '2023.6.0', 'gitdb': '4.0.10', 'google-api-core': '2.11.1', 
'google-auth': '2.22.0', 'google-auth-oauthlib': '1.0.0', 'google-cloud-bigquery': '3.11.4', 'google-cloud-bigquery-storage': '2.22.0', 'google-cloud-core': '2.3.3', 
'google-cloud-storage': '2.10.0', 'google-crc32c': '1.5.0', 'google-resumable-media': '2.5.0', 'googleapis-common-protos': '1.60.0', 'greenlet': '2.0.2', 'grpcio': '1.57.0', 
'grpcio-status': '1.57.0', 'holidays': '0.31', 'httplib2': '0.19.1', 'idna': '3.4', 'importlib-resources': '6.0.1', 'iniconfig': '2.0.0', 'ipykernel': '6.25.1', 'ipython': '8.14.0', 
'ipython-genutils': '0.2.0', 'ipywidgets': '7.8.0', 'isodate': '0.6.1', 'isoduration': '20.11.0', 'jedi': '0.19.0', 'joblib': '1.3.2', 'json5': '0.9.14', 'jsonpointer': '2.4', 
'jsonschema': '4.19.0', 'jsonschema-specifications': '2023.7.1', 'jupyter-client': '8.3.1', 'jupyter-core': '5.3.1', 'jupyter-events': '0.7.0', 'jupyter-lsp': '2.2.0', 'jupyter-server':
'2.7.3', 'jupyter-server-terminals': '0.4.4', 'jupyterlab': '4.0.5', 'jupyterlab-pygments': '0.2.2', 'jupyterlab-server': '2.24.0', 'jupyterlab-widgets': '1.1.5', 'kiwisolver': '1.4.5',
'kubernetes': '18.20.0', 'loguru': '0.7.0', 'matplotlib': '3.7.2', 'matplotlib-inline': '0.1.6', 'mistune': '3.0.1', 'monotonic': '1.6', 'multidict': '6.0.4', 'nbclient': '0.8.0', 
'nbconvert': '7.8.0', 'nbformat': '5.9.2', 'nest-asyncio': '1.5.7', 'notebook': '7.0.3', 'notebook-shim': '0.2.3', 'numpy': '1.25.2', 'oauthlib': '3.2.2', 'optuna': '3.3.0', 
'overrides': '7.4.0', 'packaging': '23.1', 'pandas': '1.3.5', 'pandas-gbq': '0.19.2', 'pandocfilters': '1.5.0', 'parso': '0.8.3', 'passlib': '1.7.4', 'pexpect': '4.8.0', 'pickleshare': 
'0.7.5', 'pip': '23.2.1', 'platformdirs': '3.10.0', 'plotly': '5.16.1', 'pluggy': '1.3.0', 'prometheus-client': '0.17.1', 'prompt-toolkit': '3.0.39', 'prophet': '1.1.4', 'proto-plus': 
'1.22.3', 'protobuf': '4.24.2', 'psutil': '5.9.5', 'ptyprocess': '0.7.0', 'pure-eval': '0.2.2', 'pyarrow': '12.0.1', 'pyasn1': '0.5.0', 'pyasn1-modules': '0.3.0', 'pycparser': '2.21', 
'pydantic': '1.10.12', 'pydata-google-auth': '1.8.2', 'pyparsing': '2.4.7', 'pytest': '7.4.0', 'pytest-cov': '4.1.0', 'python-dateutil': '2.8.2', 'python-json-logger': '2.0.7', 
'python-terraform': '0.10.1', 'pytz': '2023.3', 'pyzmq': '25.1.1', 'referencing': '0.30.2', 'requests': '2.31.0', 'requests-oauthlib': '1.3.1', 'rfc3339-validator': '0.1.4', 
'rfc3986-validator': '0.1.1', 'rich': '12.6.0', 'rpds-py': '0.10.0', 'rsa': '4.9', 'scikit-learn': '1.3.0', 'scipy': '1.11.2', 'setuptools': '68.1.2', 'six': '1.16.0', 'smmap': '5.0.0',
'sniffio': '1.3.0', 'soupsieve': '2.4.1', 'sqlalchemy2-stubs': '0.0.2a35', 'sqlmodel': '0.0.8', 'stack-data': '0.6.2', 'tenacity': '8.2.3', 'terminado': '0.17.1', 'threadpoolctl': 
'3.2.0', 'tinycss2': '1.2.1', 'tomli': '2.0.1', 'tornado': '6.3.3', 'tqdm': '4.66.1', 'traitlets': '5.9.0', 'typing-extensions': '4.7.1', 'uri-template': '1.3.0', 'urllib3': '1.26.16', 
'validators': '0.18.2', 'wcwidth': '0.2.6', 'webcolors': '1.13', 'webencodings': '0.5.1', 'websocket-client': '1.6.2', 'wheel': '0.40.0', 'widgetsnbextension': '3.6.5', 'yarl': '1.9.2',
'zenml': '0.43.0'}



### What happened?

The Kubernetes integration requires 18.20.0 (as stated [here](https://github.com/zenml-io/zenml/blob/2ca04b94518d12e10644ca113030fc168071cbc9/src/zenml/integrations/kubernetes/__init__.py#L33)).
To create a scheduled pipeline (i.e. a K8s `CronJob`), the current ZenML code uses the K8s client `BatchV1Api` to call the `create_namespaced_cron_job()` method.
However, in K8s client 18.20.0 this method is only available in the `BatchV1beta1Api` (see [here](https://github.com/kubernetes-client/python/blob/v18.20.0/kubernetes/docs/BatchV1beta1Api.md)) and not in the `BatchV1Api` (see [here](https://github.com/kubernetes-client/python/blob/v18.20.0/kubernetes/docs/BatchV1Api.md)) 
That leads to the following error ```AttributeError: 'BatchV1Api' object has no attribute 'create_namespaced_cron_job'```

Suggested fix:
- **Option 1**: use the `k8s_client.BatchV1beta1Api(self.get_kube_client())` instead of [k8s_client.BatchV1Api(self.get_kube_client())](https://github.com/zenml-io/zenml/blob/2ca04b94518d12e10644ca113030fc168071cbc9/src/zenml/integrations/kubernetes/orchestrators/kubernetes_orchestrator.py#L131)
- **Option 2**: upgrade the integration so that it requires a newer K8s Python client ( >= v21.7.0 since [CronJob are in stable version since K8s V1.21](https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/)) and update the `manifest_utils.py` so that the ` build_cron_job_manifest()` method returns a `k8s_client.V1CronJob`

What do you think ?

### 

### Reproduction steps

1. use ZenML 0.43.0 with `kubernetes` integration
2. creating a scheduled pipeline with the Kubernetes Orchestrator
will raise `AttributeError: 'BatchV1Api' object has no attribute 'create_namespaced_cron_job'`



### Relevant log output

```shell
│                                                                                                  │
│ /home/xxxx/.pyenv/versions/easyplan/lib/python3.10/site-packages/zenml/new/pipelines/pipelin │
│ e.py:1221 in __call__                                                                            │
│                                                                                                  │
│   1218 │   │   │   return self.entrypoint(*args, **kwargs)                                       │
│   1219 │   │                                                                                     │
│   1220 │   │   self.prepare(*args, **kwargs)                                                     │
│ ❱ 1221 │   │   return self._run(**self._run_args)                                                │
│   1222 │                                                                                         │
│   1223 │   def _call_entrypoint(self, *args: Any, **kwargs: Any) -> None:                        │
│   1224 │   │   """Calls the pipeline entrypoint function with the given arguments.               │
│                                                                                                  │
│ /home/xxxx/.pyenv/versions/easyplan/lib/python3.10/site-packages/zenml/new/pipelines/pipelin │
│ e.py:637 in _run                                                                                 │
│                                                                                                  │
│    634 │   │   │   # unexpected behavior                                                         │
│    635 │   │   │   constants.SHOULD_PREVENT_PIPELINE_EXECUTION = True                            │
│    636 │   │   │   try:                                                                          │
│ ❱  637 │   │   │   │   stack.deploy_pipeline(deployment=deployment_model)                        │
│    638 │   │   │   finally:                                                                      │
│    639 │   │   │   │   constants.SHOULD_PREVENT_PIPELINE_EXECUTION = False                       │
│    640                                                                                           │
│                                                                                                  │
│ /home/xxxx/.pyenv/versions/easyplan/lib/python3.10/site-packages/zenml/stack/stack.py:871 in │
│ deploy_pipeline                                                                                  │
│                                                                                                  │
│    868 │   │   Returns:                                                                          │
│    869 │   │   │   The return value of the call to `orchestrator.run_pipeline(...)`.             │
│    870 │   │   """                                                                               │
│ ❱  871 │   │   return self.orchestrator.run(deployment=deployment, stack=self)                   │
│    872 │                                                                                         │
│    873 │   def _get_active_components_for_step(                                                  │
│    874 │   │   self, step_config: "StepConfiguration"                                            │
│                                                                                                  │
│ /home/xxxx/.pyenv/versions/easyplan/lib/python3.10/site-packages/zenml/orchestrators/base_or │
│ chestrator.py:166 in run                                                                         │
│                                                                                                  │
│   163 │   │                                                                                      │
│   164 │   │   environment = get_config_environment_vars()                                        │
│   165 │   │   try:                                                                               │
│ ❱ 166 │   │   │   result = self.prepare_or_run_pipeline(                                         │
│   167 │   │   │   │   deployment=deployment, stack=stack, environment=environment                │
│   168 │   │   │   )                                                                              │
│   169 │   │   finally:                                                                           │
│                                                                                                  │
│ /home/20001285/.pyenv/versions/easyplan/lib/python3.10/site-packages/zenml/integrations/kubernet │
│ es/orchestrators/kubernetes_orchestrator.py:414 in prepare_or_run_pipeline                       │
│                                                                                                  │
│   411 │   │   │   │   mount_local_stores=self.config.is_local,                                   │
│   412 │   │   │   )                                                                              │
│   413 │   │   │                                                                                  │
│ ❱ 414 │   │   │   self._k8s_batch_api.create_namespaced_cron_job(                                │
│   415 │   │   │   │   body=cron_job_manifest,                                                    │
│   416 │   │   │   │   namespace=self.config.kubernetes_namespace,                                │
│   417 │   │   │   )                                                                              │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
AttributeError: 'BatchV1Api' object has no attribute 'create_namespaced_cron_job'
```


### Code of Conduct

- [X] I agree to follow this project's Code of Conduct


Discussion:
Hi @francoisserra, thanks for reporting this, great catch!

Both options you suggested sound good to me. Option 1 seems much quicker to implement, but Option 2 is probably the cleaner approach. Option 2 has a caveat though, which is that `kfp` requires `kubernetes>=8.0.0,<26`, so we cannot update to the newest `kubernetes` version at this moment. We could update the version range to `kubernetes>=21.7,<26` though, I guess that should work.

Would you perhaps be open to contributing a PR with a fix of your choice for this issue?

Resolving patch:
diff --git a/src/zenml/integrations/kubernetes/__init__.py b/src/zenml/integrations/kubernetes/__init__.py
--- a/src/zenml/integrations/kubernetes/__init__.py
+++ b/src/zenml/integrations/kubernetes/__init__.py
@@ -30,7 +30,7 @@ class KubernetesIntegration(Integration):
     """Definition of Kubernetes integration for ZenML."""
 
     NAME = KUBERNETES
-    REQUIREMENTS = ["kubernetes==18.20.0"]
+    REQUIREMENTS = ["kubernetes>=21.7,<26"]
 
     @classmethod
     def flavors(cls) -> List[Type[Flavor]]:
diff --git a/src/zenml/integrations/kubernetes/orchestrators/manifest_utils.py b/src/zenml/integrations/kubernetes/orchestrators/manifest_utils.py
--- a/src/zenml/integrations/kubernetes/orchestrators/manifest_utils.py
+++ b/src/zenml/integrations/kubernetes/orchestrators/manifest_utils.py
@@ -210,7 +210,7 @@ def build_cron_job_manifest(
     service_account_name: Optional[str] = None,
     env: Optional[Dict[str, str]] = None,
     mount_local_stores: bool = False,
-) -> k8s_client.V1beta1CronJob:
+) -> k8s_client.V1CronJob:
     """Create a manifest for launching a pod as scheduled CRON job.
 
     Args:
@@ -245,9 +245,9 @@ def build_cron_job_manifest(
         mount_local_stores=mount_local_stores,
     )
 
-    job_spec = k8s_client.V1beta1CronJobSpec(
+    job_spec = k8s_client.V1CronJobSpec(
         schedule=cron_expression,
-        job_template=k8s_client.V1beta1JobTemplateSpec(
+        job_template=k8s_client.V1JobTemplateSpec(
             metadata=pod_manifest.metadata,
             spec=k8s_client.V1JobSpec(
                 template=k8s_client.V1PodTemplateSpec(
@@ -258,9 +258,9 @@ def build_cron_job_manifest(
         ),
     )
 
-    job_manifest = k8s_client.V1beta1CronJob(
+    job_manifest = k8s_client.V1CronJob(
         kind="CronJob",
-        api_version="batch/v1beta1",
+        api_version="batch/v1",
         metadata=pod_manifest.metadata,
         spec=job_spec,
     )


Test patch:
diff --git a/tests/unit/integrations/kubernetes/orchestrators/test_kubernetes_orchestrator.py b/tests/unit/integrations/kubernetes/orchestrators/test_kubernetes_orchestrator.py
--- a/tests/unit/integrations/kubernetes/orchestrators/test_kubernetes_orchestrator.py
+++ b/tests/unit/integrations/kubernetes/orchestrators/test_kubernetes_orchestrator.py
@@ -82,7 +82,7 @@ def mock_load_incluster_config() -> None:
     )
 
     mocker.patch("kubernetes.client.CoreV1Api")
-    mocker.patch("kubernetes.client.BatchV1beta1Api")
+    mocker.patch("kubernetes.client.BatchV1Api")
     mocker.patch("kubernetes.client.RbacAuthorizationV1Api")
 
 
diff --git a/tests/unit/integrations/kubernetes/orchestrators/test_manifest_utils.py b/tests/unit/integrations/kubernetes/orchestrators/test_manifest_utils.py
--- a/tests/unit/integrations/kubernetes/orchestrators/test_manifest_utils.py
+++ b/tests/unit/integrations/kubernetes/orchestrators/test_manifest_utils.py
@@ -15,8 +15,8 @@
 
 import pytest
 from kubernetes.client import (
-    V1beta1CronJob,
-    V1beta1CronJobSpec,
+    V1CronJob,
+    V1CronJobSpec,
     V1ObjectMeta,
     V1Pod,
     V1PodSpec,
@@ -123,7 +123,7 @@ def test_build_cron_job_manifest_pod_settings(
     kubernetes_settings: KubernetesOrchestratorSettings,
 ):
     """Test that the pod settings are correctly set in the manifest."""
-    manifest: V1beta1CronJob = build_cron_job_manifest(
+    manifest: V1CronJob = build_cron_job_manifest(
         cron_expression="* * * * *",
         pod_name="test_name",
         run_name="test_run",
@@ -134,8 +134,8 @@ def test_build_cron_job_manifest_pod_settings(
         settings=kubernetes_settings,
         service_account_name="test_sa",
     )
-    assert isinstance(manifest, V1beta1CronJob)
-    assert isinstance(manifest.spec, V1beta1CronJobSpec)
+    assert isinstance(manifest, V1CronJob)
+    assert isinstance(manifest.spec, V1CronJobSpec)
     job_pod_spec = manifest.spec.job_template.spec.template.spec
     assert (
         job_pod_spec.affinity["nodeAffinity"][