Repository: elyra-ai/elyra

Issue:
make generic pipelines to DAG rendering work with Airflow 2.x
**Describe the issue**
Currently, even when using generic pipeline editor with python files and notebooks, the rendered DAG is not compatible with Airflow 2..6.0 or higher, or any Airflow 2.x, for that matter :-)
`
Broken DAG: [/opt/airflow/dags/repo/SvensWetterapprootpvc-0202162036.py] Traceback (most recent call last): File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed File "/opt/airflow/dags/repo/SvensWetterapprootpvc-0202162036.py", line 2, in <module> from airflow.contrib.kubernetes.volume_mount import VolumeMount ModuleNotFoundError: No module named 'airflow.contrib.kubernetes'`

**To Reproduce**
Steps to reproduce the behavior:
1. Create a generic pipeline
2. Submit it to the Airflow runtime with Airflow 2.x

**Screenshots or log output**
If applicable, add screenshots or log output to help explain your problem.
<details><summary>Log Output</summary>
<pre>
Paste the log output here.
</pre>
</details>

**Expected behavior**
DAg is rendered correctly for use in Airflow 2.x

Starting with jinja template
https://github.com/elyra-ai/elyra/blob/main/elyra/templates/airflow/airflow_template.jinja2

and elyra/elyra/pipeline/airflow/processor_airflow.py

possible hints
https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes/stable/operators.html

https://airflow.apache.org/docs/apache-airflow/2.5.3/howto/upgrading-from-1-10/index.html

https://airflow.apache.org/docs/apache-airflow/2.5.3/core-concepts/dags.html

https://github.com/apache/airflow/pull/21653

**Deployment information**
Describe what you've deployed and how:
 - Elyra version: latest
 - Installation source: [e.g. PyPI, conda, from source, official container image, custom container image]
 - Deployment type: Open Data Hub pre-RedHat takeover
 - Operating system: [e.g. macos, linux]


Discussion:
ianonavy: One thing I forgot to note is that `from airflow.providers.cncf.kubernetes.operators.kubernetes_pod import KubernetesPodOperator` was deprecated in favor of `from airflow.providers.cncf.kubernetes.operators.pod import KubernetesPodOperator` in Airflow 2.6.0. Decided to keep the older one for compatibility with earlier releases; not sure which version we are targeting.
shalberd: Regarding KubernetesPodOperator location: Yes, [good point.](https://github.com/apache/airflow/blob/v2-6-stable/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py#L25) Well, 2.5.x is already well past, plus there were some security bugs in 2.5.x that someone in our org mentioned that make it a good idea to go further than 2.5.x. Airflow 2.6.0 came out in May, so we can assume it is current.
giladd123: I did approximatly the same thing in an air tight environment so it would be a problem for me to commit my code but I think I can help.
I also encountered a problem with the resources, what seemed to work for me is switching `resources = ... ` to something like:
```
{% if operation.cpu_request or operation.mem_request or operation.gpu_limit %}
container_resources=k8s.V1ResourceRequirements(
    requests={
        {% if operation.cpu_request %}
        'cpu': '{{ operation.cpu_request }}',
        {% endif %}
        {% if operation.mem_request %}
        'memory': '{{ operation.mem_request }}G',
        {% endif %}
    },
    limits={
        {% if operation.cpu_request %}
        'cpu': '{{ operation.cpu_request }}',
        {% endif %}
        {% if operation.mem_request %}
        'memory': '{{ operation.mem_request }}G',
        {% endif %}
        {% if operation.gpu_limit %}
        'nvidia.com/gpu': '{{ operation.gpu_limit }}', # or 'amd.com/gpu'
        {% endif %}
    },
)
{% endif %}
```
In this case, the UI doesn't have an option for CPU and Memory limits, so the limits could probably be removed (In my k8s cluster a limit must be set so I just set it in some ratio of the request).

I lack the knowledge to change the UI to include limits but I'm willing to help with anything else!
shalberd: @giladd123 aggreed, setting limits is good practice, even when not enforced by [LimitRange](https://docs.openshift.com/container-platform/4.10/nodes/clusters/nodes-cluster-limit-ranges.html#nodes-cluster-limit-ranges-limits_nodes-cluster-limit-ranges) and max ratio request to limit.

[Quoting](https://www.opensourcerers.org/2022/06/20/the-importance-of-getting-resource-requests-and-limits-right/)

```
Requests and Limits should never be treated as min/max requirements for any pod. The requested amount should be a reflection of the pod’s required resources (both CPU and memory) under load. The ratio between request and limit should be as close to 1.0 as possible. Any divergence from this should have a valid business reason. The request amount reflects the pod’s normal usage, the limit is there as a safety net to protect the rest of the cluster’s load, in case it misbehaves, or a mistake was made during the profiling of the application.
```
For the template, setting limits to either equal to requests or minimally higher by a factor of x 1.2 would be good practice and make the cluster as a whole more stable. In the end, developers sets those requests in the GUI and they must be admonished to set realistic request sizes. With a limit set, they'll realize soon they are wrong in their pipeline steps definition for resources.

giladd123: > For the template, setting limits to either equal to requests or minimally higher by a factor of x 1.2 would be good practice and make the cluster as a whole more stable.

Altough being a good practice, I don't think setting a hard ratio is a good idea as in the end this gives the user less flexability (for example, my cluster enforces a 1:4 ratio on cpu request and limit, with a 1.2x jobs will just not go up). I don't think we should force good practices on users, we should let them deal with this themselves.
shalberd: @giladd123 Agreed, however, not setting limits at all leads to pods being scheduled on a node that they really should not be scheduled on. In your example of 1 to 4 ratio, should all pods every run on the same node at the same time, you're in for huge trouble, having a node outage and re-scheduling, leading to interruptions.

As you mentioned, it'd be best to at least have the option to get limits at the GUI level. 
Then, each enterprise can make their developers input educated settings. 
I need to check whether cpu request, memory request, gpu request, gpu memory request are GUI required values. 
If they are, then the same (mandatory fields) should apply for limits on the GUI as well, being of course user-defineable.
Not providing limits is simply sloppy, I've got no idea why noone ever criticized that.
lresende: Where we are with this? I am assuming still a WIP and we are waiting for more changes before a full review?
dolevi101: Hi, is there any update?
shalberd: yes, it is WIP still, in large part due to the fact that I have not received any feedback, neither from ODH nor here, on why CPU and Memory limits are not in the variables and thus cannot be used at least optionally. 
shalberd: I am leaving out the issue of resource limits for now. My aim is to test build this and integrate it into an open data hub notebook container, then test it with our Airflow 2.6.2 instance that is reading the Elyra-generated DAGs from our Gitlab company-internally.
shalberd: getting there. Muuch better. Just tested this an forgot about what Gilad told me back then regarding changes to resources assembly, 
![Bildschirmfoto 2023-11-24 um 17 11 29](https://github.com/elyra-ai/elyra/assets/21118431/a7ca59f6-43b4-4c3e-9f41-afd439555976)
![Bildschirmfoto 2023-11-24 um 17 11 53](https://github.com/elyra-ai/elyra/assets/21118431/ef7df11d-87d2-4524-99f9-970803aee239)

https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes/stable/_modules/airflow/providers/cncf/kubernetes/operators/pod.html

making the changes now and will test the locally-built wheel file once more with Airflow 2.6.x then, next week.

i.e. in the airflow template file, for now only with that gpu limit, no cpu and memory limits as we do not have them yet in GUI.
```
        {% if operation.cpu_request or operation.mem_request or operation.gpu_limit %}
                                                            container_resources=k8s.V1ResourceRequirements(
                                                                requests={
                                                                    {% if operation.cpu_request %}
                                                                        'cpu': '{{ operation.cpu_request }}',
                                                                    {% endif %}
                                                                    {% if operation.mem_request %}
                                                                        'memory': '{{ operation.mem_request }}G',
                                                                    {% endif %}
                                                                },
                                                                limits={
                                                                    {% if operation.gpu_limit %}
                                                                        'nvidia.com/gpu': '{{ operation.gpu_limit }}', # or 'amd.com/gpu'
                                                                    {% endif %}
                                                                },
                                                            )
        {% endif %}
```
I don't use GPUs, but I want to find out before I make the whl again whether there is also a property for the kind of gpu, don't wanna hard-code either `nvidia.com/gpu` or `amd.com/gpu`
Another thing I noticed: CPU always needs to be an integer greater than 1, memory also. This is not good. We should later fix that GUI and property part to more K8S and Openshift style resource units: https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/
shalberd: squashed commits as well to be more readable
lresende: It would be good to have a 2nd person to validate this is working before we look into merging it... 
kevin-bates: @shalberd - thank you for this pull request.  I am surprised at how little changes there are and don't see any kind of Airflow version checking happening.  With these changes, is elyra agnostic to the Airflow version or do these changes _replace_ the existing Airflow 1x support with Airflow 2x support?  I guess I was expecting to see a new subclass (e.g., `AirflowPipelineProcessor2`) or runtime type (`APACHE_AIRFLOW_2`) so as to preserve existing support.  If a replacement - what do we tell our existing Airflow 1x users?
shalberd: @kevin-bates it is a replacement for Airflow 2.x. I have already talked with @lresende in a Friday community call and we have the idea so far to make this part of Elyra 4.x, i.e. no more Airflow 1.x support soon. About lifecycle management: I only have contact with Astronomer and the Airflow community helm chart maintainer @thesuperzapper ... my judgment call would be: Airflow 1.x is long deprecated, no more security updates ... announce Airflow 1.x support derelease with Elyra 4.x
MR-GOYAL: @shalberd I have tried changes as mentioned in the [fork](https://github.com/shalberd/elyra/tree/make_generic_pipeline_work_with_airflow2x) and its working fine with the airflow 2.7.2(version we are using). I have tested in our environment and airflow pipeline are executing as expected.
Thanks for the airflow 2.x support on elyra
romeokienzler: waiting for #3202 to be merged
lresende: Looks great that the generic pipelines are working with these changes, we need to make sure custom components made for Airflow 2.x are also working well.  
shalberd: confirming this here is still working beautifully for Airflow 2.8.2
shalberd: @lresende the readthedocs latest: I searched for all mentions of Airflow there and made some changes to the documentation, pertaining to Airflow 2 support vs. Airflow 1 support, the scope of Airflow 2 support in Elyra 4.
shalberd: Hi @lresende I had to fix some lint-server E226 missing whitespace around arithmetic operator code in elyra/tests, but all well now.

shalberd: @lresende getting an error at step validate runtime images 

"=> ERROR: Container image pytorch/pytorch:2.5.1-cuda11.8-cudnn9-devel is not a suitable Elyra runtime image
make[1]: *** [Makefile:326: validate-runtime-image] Error 1"
shalberd: /retest
shalberd: @caponetto looking much better, your change worked for the validate images step. Nice job, thanks again.

You mentioned that cypress component, do you know why this check fails in the step "run integration tests":

<img width="1505" alt="Screenshot 2024-12-20 at 20 57 19" src="https://github.com/user-attachments/assets/553dc95f-db45-4e1c-bf2d-def6593a5520" />

caponetto: @shalberd From my observations over the past few months, I believe there is a flaky integration test. Sometimes it passes, sometimes it fails. You can try rerunning the job a few times to see if it passes.
shalberd: > From my observations over the past few months, I believe there is a flaky integration test. Sometimes it passes, sometimes it fails. You can try rerunning the job a few times to see if it passes.

Yes, I amended my commit with nothing and force-pushed again, leading to the tests to run through again (I do not have enough rights to re-run checks from the Github GUI).

All tests, including that flaky yarn cypress integration test, ran through now. Wonder if we should have a separate issue filed for that flaky yarn cy integration test ....
shalberd: About other people running this Airflow 2 specific code to generate DAG python code:

Another person from India, thanks, @MR-GOYAL, also verified earlier this year with an earlier Airflow version that it worked fine. https://github.com/elyra-ai/elyra/pull/3167#issuecomment-1873860868

caponetto: > Wonder if we should have a separate issue filed for that flaky yarn cy integration test

I believe so. Also, if the failure is too frequent to the point of slowing work down, I suggest disabling it until its stability is restored. 
shalberd: Hi @lresende I rebased my feature branch with the latest changes to main, i.e. your refactor of [pipeline implementation and namings](https://github.com/elyra-ai/elyra/pull/3160) It should not have any impact on my code changes here.

Patch:
diff --git a/docs/source/getting_started/changelog.md b/docs/source/getting_started/changelog.md
index eb8045b7..954fe70e 100644
--- a/docs/source/getting_started/changelog.md
+++ b/docs/source/getting_started/changelog.md
@@ -4,7 +4,11 @@ A summary of new feature highlights is located on the [GitHub release page](http
 
 ## Release 4.0.0
 
-- Add functionality to select resource limits from the GUI [#3202](https://github.com/elyra-ai/elyra/pull/3202)
+- Add support for JupyterLab 4.x - [#3201](https://github.com/elyra-ai/elyra/pull/3201)
+- Add functionality to select resource limits from the GUI - [#3202](https://github.com/elyra-ai/elyra/pull/3202)
+- Separate KFP and Airflow libraries during build step, making kfp dependency optional for non-KFP runtime users - [#3248](https://github.com/elyra-ai/elyra/pull/3248)
+- KFP and Airflow runtime generic components logging during execution and S3 file put for run log output file made configurable - [#3227](https://github.com/elyra-ai/elyra/pull/3227) 
+- Generic pipelines and generic components runtime support changed to Airflow >= 2.7.0. Airflow 1.x no longer supported - [#3167](https://github.com/elyra-ai/elyra/pull/3167)
 
 ## Release 3.15.0 - 03/28/2023
 
diff --git a/docs/source/getting_started/tutorials.md b/docs/source/getting_started/tutorials.md
index f24b2a27..6ee9bd8c 100644
--- a/docs/source/getting_started/tutorials.md
+++ b/docs/source/getting_started/tutorials.md
@@ -38,10 +38,12 @@ Learn how to [run runtime-specific pipelines on Kubeflow Pipelines](https://gith
 #### Run generic pipelines on Apache Airflow
 
 Learn how to [run generic pipelines on Apache Airflow](https://github.com/elyra-ai/examples/tree/main/pipelines/run-generic-pipelines-on-apache-airflow). This tutorial requires an Apache Airflow deployment in a local environment or on the cloud.
+If you want to run generic pipelines in Airflow >= 2.7.0, you have to use Elyra 4. In Elyra 4, generic pipeline support for Airflow 1 is removed.
 
 #### Run runtime-specific pipelines on Apache Airflow
 
 Learn how to [run runtime-specific pipelines on Apache Airflow](https://github.com/elyra-ai/examples/tree/main/pipelines/run-pipelines-on-apache-airflow). This tutorial requires an Apache Airflow deployment in a local environment or on the cloud.
+If you want to run generic components (R, Python, ipynb Notebooks) in runtime-specific pipelines in Airflow >= 2.7.0, you have to use Elyra 4. In Elyra 4, generic pipeline support for Airflow 1 is removed and custom Airflow components are not yet supported.
 
 
 #### Examples
diff --git a/docs/source/recipes/configure-airflow-as-a-runtime.md b/docs/source/recipes/configure-airflow-as-a-runtime.md
index be70d466..1b6b91d1 100644
--- a/docs/source/recipes/configure-airflow-as-a-runtime.md
+++ b/docs/source/recipes/configure-airflow-as-a-runtime.md
@@ -24,6 +24,8 @@ Pipelines in Elyra can be run locally in JupyterLab, or remotely on Kubeflow Pip
 **Note: Support for Apache Airflow is experimental.**
 
 This document outlines how to set up a new Elyra-enabled Apache Airflow environment or add Elyra support to an existing deployment.
+You can submit pipelines with generic components to Airflow >= 2.7.0 from Elyra 4 on.
+Generic components DAG code generation support for Airflow 1.x is removed in Elyra 4.
   
 This guide assumes a general working knowledge of and administration of a Kubernetes cluster.
 
@@ -42,8 +44,10 @@ AND
 OR  
   
 - An existing Apache Airflow cluster 
-    - Ensure Apache Airflow is at least v1.10.8 and below v2.0.0. Other versions might work but have not been tested.
+    - Ensure Apache Airflow is at least v1.10.8 and below v2.0.0. This applies to Elyra < 4.
+    - Ensure Apache Airflow is at least v2.7.0. This applies to Elyra 4.
     - Apache Airflow is configured to use the Kubernetes Executor.
+    - Apache Airflow must be configured to use git-sync, which is configurable both in [Airflow 1](https://airflow.apache.org/docs/apache-airflow/1.10.12/configurations-ref.html?highlight=git%20sync#git-repo) as well as in [Airflow 2](https://airflow.apache.org/docs/helm-chart/stable/parameters-ref.html#airflow) 
     - Ensure the KubernetesPodOperator is installed and available in the Apache Airflow deployment
     
 ## Setting up a DAG repository on Git
diff --git a/docs/source/user_guide/best-practices-custom-pipeline-components.md b/docs/source/user_guide/best-practices-custom-pipeline-components.md
index 1bc3ee4a..3ea6dcad 100644
--- a/docs/source/user_guide/best-practices-custom-pipeline-components.md
+++ b/docs/source/user_guide/best-practices-custom-pipeline-components.md
@@ -40,6 +40,8 @@ See the [Kubeflow Pipelines documentation](https://www.kubeflow.org/docs/compone
 ### Apache Airflow components
 
 #### Requirements
+Apache Airflow components are currently only supported for Airflow < 2 and in Elyra < 4.
+Elyra 4 starts with generic components support (R, Python, ipynb Notebooks), not (yet) for custom components, for Airflow >= 2.7.0.
 
 ##### Configure fully qualified package names for custom operator classes
 
@@ -161,4 +163,4 @@ The missing component definition is stored in a [Machine Learning Exchange](http
 
 #### Component catalogs not listed here
 
-Check the [component catalog connector directory](https://github.com/elyra-ai/examples/blob/main/component-catalog-connectors/connector-directory.md) if the referenced catalog type is not listed here.
\ No newline at end of file
+Check the [component catalog connector directory](https://github.com/elyra-ai/examples/blob/main/component-catalog-connectors/connector-directory.md) if the referenced catalog type is not listed here.
diff --git a/docs/source/user_guide/pipeline-components.md b/docs/source/user_guide/pipeline-components.md
index e8cfc306..9603d029 100644
--- a/docs/source/user_guide/pipeline-components.md
+++ b/docs/source/user_guide/pipeline-components.md
@@ -24,11 +24,11 @@ limitations under the License.
 
 ![A basic Kubeflow pipeline](../images/user_guide/pipeline-components/kubeflow-pipeline.png)
 
-The same pipeline could be implemented using a single component that performs all these tasks, but that component might not be as universally re-usable. Consider, for example, that for another project the data resides in a different kind of storage. With fine-granular components you'd only have to replace the load data component with one that supports the other storage type and could retain everything else. 
+The same pipeline could be implemented using a single component that performs all these tasks, but that component might not be as universally re-usable. Consider, for example, that for another project the data resides in a different kind of storage. With fine-granular components you'd only have to replace the load data component with one that supports the other storage type and could retain everything else.
 
 #### Generic components
 
-Elyra includes three _generic components_ that allow for the processing of Jupyter notebooks, Python scripts, and R scripts. These components are called generic because they can be included in pipelines for any supported runtime type: local/JupyterLab, Kubeflow Pipelines, and Apache Airflow. Components are exposed in the pipeline editor via the palette.
+Elyra includes three _generic components_ that allow for the processing of Jupyter notebooks, Python scripts, and R scripts. These components are called generic because they can be included in pipelines for any supported runtime type: local/JupyterLab, Kubeflow Pipelines, and Apache Airflow >= 2.7.0. Components are exposed in the pipeline editor via the palette.
 
 ![Generic components in the palette](../images/user_guide/pipeline-components/generic-components-in-palette.png)
 
@@ -36,7 +36,7 @@ Note: Refer to the [_Best practices_ topic in the _User Guide_](best-practices-f
 
 #### Custom components
 
-_Custom components_ are commonly only implemented for one runtime type, such as Kubeflow Pipelines or Apache Airflow. (The local runtime type does not support custom components.)
+_Custom components_ are commonly only implemented for one runtime type, such as Kubeflow Pipelines or Apache Airflow < 2. (The local runtime type does not support custom components). Custom components for Apache Airflow, due to their being supported only for Airflow 1.x, are only supported on Elyra < 4.
 
 ![Kubeflow components in the palette](../images/user_guide/pipeline-components/custom-kubeflow-components-in-palette.png)
 
@@ -63,9 +63,9 @@ Elyra includes connectors for the following component catalog types:
 
     Example: A URL component catalog that is configured using the `http://myserver:myport/mypath/my_component.yaml` URL makes the `my_component.yaml` component file available to Elyra.
 
- - [_Apache Airflow package catalogs_](#apache-airflow-package-catalog) provide access to Apache Airflow operators that are stored in Apache Airflow built distributions.
+ - [_Apache Airflow package catalogs_](#apache-airflow-package-catalog) provide access to Apache Airflow operators that are stored in Apache Airflow built distributions. This is currently only supported for Airflow < 2.
 
- - [_Apache Airflow provider package catalogs_](#apache-airflow-provider-package-catalog) provide access to Apache Airflow operators that are stored in Apache Airflow provider packages.  
+ - [_Apache Airflow provider package catalogs_](#apache-airflow-provider-package-catalog) provide access to Apache Airflow operators that are stored in Apache Airflow provider packages. This is currently only supported for Airflow < 2.  
 
 Refer to section [Built-in catalog connector reference](#built-in-catalog-connector-reference) for details about these connectors. 
 
@@ -438,6 +438,7 @@ Examples (CLI):
 
 The [Apache Airflow package catalog connector](https://github.com/elyra-ai/elyra/tree/main/elyra/pipeline/airflow/package_catalog_connector) provides access to operators that are stored in Apache Airflow [built distributions](https://packaging.python.org/en/latest/glossary/#term-built-distribution):
 - Only the [wheel distribution format](https://packaging.python.org/en/latest/glossary/#term-Wheel) is supported.
+- Only Airflow < 2 is supported. Use of that functionality is not working in Elyra >=4, which is no longer supporting Airflow 1.x.
 - The specified URL must be retrievable using an HTTP `GET` request. `http`, `https`, and `file` [URI schemes](https://www.iana.org/assignments/uri-schemes/uri-schemes.xhtml) are supported.
 - In secured environments where SSL server authenticity can only be validated using certificates based on private public key infrastructure (PKI) with root and optionally intermediate certificate authorities (CAs) that are not publicly trusted, you must define environment variable `TRUSTED_CA_BUNDLE_PATH` in the environment where JupyterLab/Elyra is running. The variable value must identify an existing [Privacy-Enhanced Mail (PEM) file](https://en.wikipedia.org/wiki/Privacy-Enhanced_Mail).
 
@@ -454,6 +455,7 @@ Examples:
 #### Apache Airflow provider package catalog
 The [Apache Airflow provider package catalog connector](https://github.com/elyra-ai/elyra/tree/main/elyra/pipeline/airflow/provider_package_catalog_connector) provides access to operators that are stored in [Apache Airflow provider packages](https://airflow.apache.org/docs/apache-airflow-providers/):
 - Only the [wheel distribution format](https://packaging.python.org/en/latest/glossary/#term-Wheel) is supported.
+- Only Airflow < 2 and operators for Airflow < 2 are supported. Use of that functionality is not working in Elyra >=4, which is no longer supporting Airflow 1.x. 
 - The specified URL must be retrievable using an HTTP `GET` request. `http`, `https`, and `file` [URI schemes](https://www.iana.org/assignments/uri-schemes/uri-schemes.xhtml) are supported.
 - In secured environments where SSL server authenticity can only be validated using certificates based on private public key infrastructure (PKI) with root and optionally intermediate certificate authorities (CAs) that are not publicly trusted, you must define environment variable `TRUSTED_CA_BUNDLE_PATH` in the environment where JupyterLab/Elyra is running. The variable value must identify an existing [Privacy-Enhanced Mail (PEM) file](https://en.wikipedia.org/wiki/Privacy-Enhanced_Mail).
 
diff --git a/elyra/pipeline/airflow/airflow_processor.py b/elyra/pipeline/airflow/airflow_processor.py
index c1dc4cb0..82ab8a0b 100644
--- a/elyra/pipeline/airflow/airflow_processor.py
+++ b/elyra/pipeline/airflow/airflow_processor.py
@@ -345,6 +345,7 @@ be fully qualified (i.e., prefixed with their package names).
                     "cpu_limit": operation.cpu_limit,
                     "memory_limit": operation.memory_limit,
                     "gpu_limit": operation.gpu,
+                    "gpu_vendor": operation.gpu_vendor,
                     "operator_source": operation.filename,
                 }
 
@@ -598,13 +599,23 @@ be fully qualified (i.e., prefixed with their package names).
         str_to_render = ""
         for v in elyra_properties.get(pipeline_constants.MOUNTED_VOLUMES, []):
             str_to_render += f"""
-                Volume(name="{v.pvc_name}", configs={{"persistentVolumeClaim": {{"claimName": "{v.pvc_name}"}}}}),"""
+                k8s.V1Volume(
+                    name="{v.pvc_name}",
+                    persistent_volume_claim=k8s.V1PersistentVolumeClaimVolumeSource(
+                        claim_name="{v.pvc_name}",
+                    ),
+                ),"""
         # set custom shared memory size
         shm = elyra_properties.get(pipeline_constants.KUBERNETES_SHARED_MEM_SIZE)
         if shm is not None and shm.size:
-            config = f"""configs={{"emptyDir": {{"medium": "Memory", "sizeLimit": "{shm.size}{shm.units}"}}}}"""
             str_to_render += f"""
-                Volume(name="shm", {config}),"""
+                k8s.V1Volume(
+                    name="shm",
+                    empty_dir=k8s.V1EmptyDirVolumeSource(
+                        medium="Memory",
+                        size_limit="{shm.size}{shm.units}",
+                    ),
+                ),"""
         return dedent(str_to_render)
 
     def render_mounts(self, elyra_properties: Dict[str, ElyraProperty]) -> str:
@@ -615,8 +626,12 @@ be fully qualified (i.e., prefixed with their package names).
         str_to_render = ""
         for v in elyra_properties.get(pipeline_constants.MOUNTED_VOLUMES, []):
             str_to_render += f"""
-                 VolumeMount(name="{v.pvc_name}", mount_path="{v.path}",
-                 sub_path="{v.sub_path}", read_only={v.read_only}),"""
+                k8s.V1VolumeMount(
+                    name="{v.pvc_name}",
+                    mount_path="{v.path}",
+                    sub_path="{v.sub_path}",
+                    read_only={v.read_only},
+                ),"""
         return dedent(str_to_render)
 
     def render_secrets(self, elyra_properties: Dict[str, ElyraProperty], cos_secret: Optional[str]) -> str:
diff --git a/elyra/templates/airflow/airflow_template.jinja2 b/elyra/templates/airflow/airflow_template.jinja2
index fc68a55f..b9245695 100644
--- a/elyra/templates/airflow/airflow_template.jinja2
+++ b/elyra/templates/airflow/airflow_template.jinja2
@@ -1,15 +1,15 @@
 from airflow import DAG
-from airflow.utils.dates import days_ago
+import pendulum
 
 args = {
     'project_id' : '{{ pipeline_name }}',
 }
 
 dag = DAG(
-    '{{ pipeline_name }}',
+    dag_id='{{ pipeline_name }}',
     default_args=args,
-    schedule_interval='@once',
-    start_date=days_ago(1),
+    schedule='@once',
+    start_date=pendulum.today('UTC').add(days=-1),
     description="""
 {{ pipeline_description|replace("\"\"\"", "\\\"\\\"\\\"") }}
     """,
@@ -22,10 +22,9 @@ dag = DAG(
 {{import_statement}}
 {% endfor %}
 {% else %}
-from airflow.kubernetes.secret import Secret
-from airflow.contrib.kubernetes.volume import Volume
-from airflow.contrib.kubernetes.volume_mount import VolumeMount
-from airflow.contrib.operators.kubernetes_pod_operator import KubernetesPodOperator
+from airflow.providers.cncf.kubernetes.secret import Secret
+from kubernetes.client import models as k8s
+from airflow.providers.cncf.kubernetes.operators.pod import KubernetesPodOperator
 {% endif %}
 
 {% if operation.operator_source %}# Operator source: {{ operation.operator_source }}{% endif %}
@@ -48,23 +47,27 @@ op_{{ operation.id|regex_replace }} = KubernetesPodOperator(name='{{ operation.n
                                                             task_id='{{ operation.notebook|regex_replace }}',
                                                             env_vars={{ operation.pipeline_envs }},
         {% if operation.cpu_request or operation.mem_request or operation.cpu_limit or operation.memory_limit or operation.gpu_limit %}
-                                                            resources = {
-            {% if operation.cpu_request %}
-                                                                       'request_cpu': '{{ operation.cpu_request }}',
-            {% endif %}
-            {% if operation.mem_request %}
-                                                                       'request_memory': '{{ operation.mem_request }}G',
-            {% endif %}
-            {% if operation.cpu_limit %}
-                                                                       'limit_cpu': '{{ operation.cpu_limit }}',
-            {% endif %}
-            {% if operation.memory_limit %}
-                                                                       'limit_memory': '{{ operation.memory_limit }}G',
-            {% endif %}
-            {% if operation.gpu_limit %}
-                                                                       'limit_gpu': '{{ operation.gpu_limit }}',
-            {% endif %}
-                                                            },
+                                                            container_resources=k8s.V1ResourceRequirements(
+                                                                requests={
+                                                                    {% if operation.cpu_request %}
+                                                                        'cpu': '{{ operation.cpu_request }}',
+                                                                    {% endif %}
+                                                                    {% if operation.mem_request %}
+                                                                        'memory': '{{ operation.mem_request }}G',
+                                                                    {% endif %}
+                                                                },
+                                                                limits={
+                                                                    {% if operation.cpu_limit %}
+                                                                        'cpu': '{{ operation.cpu_limit }}',
+                                                                    {% endif %}
+                                                                    {% if operation.memory_limit %}
+                                                                        'memory': '{{ operation.memory_limit }}G',
+                                                                    {% endif %}
+                                                                    {% if operation.gpu_limit %}
+                                                                        '{{ operation.gpu_vendor }}': '{{ operation.gpu_limit }}',
+                                                                    {% endif %}
+                                                                }
+                                                            ),
         {% endif %}
                                                             volumes=[{{ processor.render_volumes(operation.elyra_props) }}],
                                                             volume_mounts=[{{ processor.render_mounts(operation.elyra_props) }}],
@@ -73,7 +76,7 @@ op_{{ operation.id|regex_replace }} = KubernetesPodOperator(name='{{ operation.n
                                                             labels={{ processor.render_labels(operation.elyra_props) }},
                                                             tolerations=[{{ processor.render_tolerations(operation.elyra_props) }}],
                                                             in_cluster={{ in_cluster }},
-                                                            config_file="{{ kube_config_path }}",
+                                                            config_file={% if kube_config_path is string %}"{{ kube_config_path }}"{% else %}{{ kube_config_path }}{% endif %},
 {% endif %}
                                                             dag=dag)
     {% if operation.image_pull_policy %}


Test patch:
diff --git a/elyra/tests/pipeline/airflow/test_processor_airflow.py b/elyra/tests/pipeline/airflow/test_processor_airflow.py
index ae7687c8..1d817877 100644
--- a/elyra/tests/pipeline/airflow/test_processor_airflow.py
+++ b/elyra/tests/pipeline/airflow/test_processor_airflow.py
@@ -195,7 +195,7 @@ def test_create_file(monkeypatch, processor, parsed_pipeline, parsed_ordered_dic
         with open(response) as f:
             file_as_lines = f.read().splitlines()
 
-        assert "from airflow.contrib.operators.kubernetes_pod_operator import KubernetesPodOperator" in file_as_lines
+        assert "from airflow.providers.cncf.kubernetes.operators.pod import KubernetesPodOperator" in file_as_lines
 
         # Check DAG project name
         for i in range(len(file_as_lines)):