Repository: kubeflow/pipelines

Issue:
Python 3 support in Dataflow component
Apache beam has had python 3 support since version 2.14.0, and python 2 will reach EOL in January 2020. Right now it looks like using `python2` is hard coded into the [Dataflow component](https://github.com/kubeflow/pipelines/blob/master/components/gcp/container/component_sdk/python/kfp_component/google/dataflow/_launch_python.py):

```python
def _prepare_cmd(project_id, python_file_path, args, staging_location):
    dataflow_args = [
        '--runner', 'dataflow', 
        '--project', project_id]
    if staging_location:
        dataflow_args += ['--staging_location', staging_location, '--temp_location', staging_location]
    return (['python2', '-u', python_file_path] + 
        dataflow_args + args)
```

Can an option be added to run Dataflow jobs using `python3`?
Python 3 support in Dataflow component
Apache beam has had python 3 support since version 2.14.0, and python 2 will reach EOL in January 2020. Right now it looks like using `python2` is hard coded into the [Dataflow component](https://github.com/kubeflow/pipelines/blob/master/components/gcp/container/component_sdk/python/kfp_component/google/dataflow/_launch_python.py):

```python
def _prepare_cmd(project_id, python_file_path, args, staging_location):
    dataflow_args = [
        '--runner', 'dataflow', 
        '--project', project_id]
    if staging_location:
        dataflow_args += ['--staging_location', staging_location, '--temp_location', staging_location]
    return (['python2', '-u', python_file_path] + 
        dataflow_args + args)
```

Can an option be added to run Dataflow jobs using `python3`?


Discussion:
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

This issue has been automatically closed because it has not had recent activity. Please comment "/reopen" to reopen it.

/reopen

This issue is still open. Beam and Dataflow will soon stop supporting python2 pipelines. After that, kubeflow may not be able launch Dataflow pipelines.

Also, this might be a duplicate of: https://github.com/kubeflow/kubeflow/issues/4894
@aaltay: You can't reopen an issue/PR unless you authored it or you are a collaborator.

<details>

In response to [this](https://github.com/kubeflow/pipelines/issues/2601#issuecomment-654370288):

>/reopen
>
>This issue is still open. Beam and Dataflow will soon stop supporting python2 pipelines. After that, kubeflow may not be able launch Dataflow pipelines.
>
>Also, this might be a duplicate of: https://github.com/kubeflow/kubeflow/issues/4894


Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
</details>
/reopen
@TheMichaelHu: Reopened this issue.

<details>

In response to [this](https://github.com/kubeflow/pipelines/issues/2601#issuecomment-654372187):

>/reopen


Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
</details>
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

Friendly ping to keep this open.

Dataflow will support only python3 before the end of the year (very likely starting in October). Beam also EOLed its py2 support and newer Beam versions starting 2.25.0 will be supporting python3 only.

My suggestion is to change:

return (['python2', '-u', python_file_path] + 
        dataflow_args + args)

to use `python3` instead of `python2`. I can make this change but I do not have how this will impact kubeflow users.
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

This issue has been automatically closed because it has not had recent activity. Please comment "/reopen" to reopen it.

/reopen

This issue is still open. Beam and Dataflow will soon stop supporting python2 pipelines. After that, kubeflow may not be able launch Dataflow pipelines.

Also, this might be a duplicate of: https://github.com/kubeflow/kubeflow/issues/4894
@aaltay: You can't reopen an issue/PR unless you authored it or you are a collaborator.

<details>

In response to [this](https://github.com/kubeflow/pipelines/issues/2601#issuecomment-654370288):

>/reopen
>
>This issue is still open. Beam and Dataflow will soon stop supporting python2 pipelines. After that, kubeflow may not be able launch Dataflow pipelines.
>
>Also, this might be a duplicate of: https://github.com/kubeflow/kubeflow/issues/4894


Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
</details>
/reopen
@TheMichaelHu: Reopened this issue.

<details>

In response to [this](https://github.com/kubeflow/pipelines/issues/2601#issuecomment-654372187):

>/reopen


Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
</details>
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

Friendly ping to keep this open.

Dataflow will support only python3 before the end of the year (very likely starting in October). Beam also EOLed its py2 support and newer Beam versions starting 2.25.0 will be supporting python3 only.

My suggestion is to change:

return (['python2', '-u', python_file_path] + 
        dataflow_args + args)

to use `python3` instead of `python2`. I can make this change but I do not have how this will impact kubeflow users.

Resolving patch:
diff --git a/components/gcp/container/component_sdk/python/kfp_component/google/dataflow/_launch_python.py b/components/gcp/container/component_sdk/python/kfp_component/google/dataflow/_launch_python.py
--- a/components/gcp/container/component_sdk/python/kfp_component/google/dataflow/_launch_python.py
+++ b/components/gcp/container/component_sdk/python/kfp_component/google/dataflow/_launch_python.py
@@ -97,7 +97,7 @@ def _prepare_cmd(project_id, python_file_path, args, staging_location):
         '--project', project_id]
     if staging_location:
         dataflow_args += ['--staging_location', staging_location, '--temp_location', staging_location]
-    return (['python2', '-u', python_file_path] + 
+    return (['python', '-u', python_file_path] + 
         dataflow_args + args)
 
 def _extract_job_id_and_location(line):
@@ -114,4 +114,4 @@ def _install_requirements(requirements_file_path):
     if not requirements_file_path:
         return
     requirements_file_path = stage_file(requirements_file_path)
-    subprocess.check_call(['pip2', 'install', '-r', requirements_file_path])
\ No newline at end of file
+    subprocess.check_call(['pip', 'install', '-r', requirements_file_path])
\ No newline at end of file


Test patch: