Repository: certbot/certbot
Pull request: Correct certbot-auto for Fedora 29+
Fixes #6698 

Fedora maintainers engaged a deprecation path for Python 2.x with Fedora 29. As a first step, `python2-virtualenv` does not install the `virtualenv` binary anymore, in favor of `python3-virtualenv`, and so the installation of Python 3 virtual environments by default.

However, certbot-auto installs `python2-virtualenv` for all recent RPM distributions, and relies of the execution of `virtualenv`, and this is failing the process.

Since the plan in the future is to remove Python 2.x from Fedora, this PR follows this logic to fix certbot-auto: started to Fedora 29, certbot-auto will install and execute certbot on Python 3. This implies to detect that we are on Fedora 29+, install `python3-virtualenv` that will install also Python 3 dependencies and `virtualenv` binary, then instruct the process to use Python 3. This is in fact similar to EOL distributions shipping with Python 2.6, and for which Python 3.4 from EPEL is installed and used.

Older versions of Fedora continue to use Python 2.x, and their process is untouched. Four scenarios are covered here:
* fresh Fedora 28: old process is used, nothing changes
* fresh Fedora 29: new process is used, Python 3 is installed, certbot runs on it
* update Fedora 29 from 28, already installed certbot-auto without rebootstrapping required: existing venv continue to be used, certbot runs on it
* update Fedora 29 from 28, already installed certbot-auto with rebootstrapping required: new process is used, installing `python3-virtualenv`, `python3-devel` and `python3-rpm-macros`, Python 3 is installed, certbot runs on it



Problem statement:
certbot-auto doesn't work on Fedora 29
## My operating system is (include version):

Fedora 29

## I installed Certbot with (certbot-auto, OS package manager, pip, etc):

certbot-auto

## I ran this command and it produced this output:

```
# git checkout v0.30.0
# ./certbot-auto -n
<bunch of stuff>
Creating virtual environment...
./certbot-auto: line 1003: virtualenv: command not found
```

## Certbot's behavior differed from what I expected because:

`virtualenv` should probably be found.

Hints from issue discussion:
Is this a regression?
Kind of? `certbot-auto` did used to work on Fedora and we're still passing tests on an old version in our test farm tests. I think what probably happened here is a more recent Fedora version changed the name of the virtualenv package and we haven't yet updated certbot-auto to reflect that.

I think we should fix this and I think doing so is probably pretty easy, however, I don't think it's a super high priority because we have very few Fedora users. As far as I know, no users have reported this bug yet.
I mean, is this a regression based on our recent changes in response to the `pip` issues.
Oh. No it is not.
> /usr/bin/virtualenv is now in the python3-virtualenv package
> The virtualenv command now comes from the python3-virtualenv package, as opposed to earlier releases where the command was in the python2-virtualenv. This effectively switches the command to Python 3; if you run virtualenv without any additional options, it will create Python 3 environments. Use virtualenv -p python2.7 to get the previously default behavior.

https://docs.fedoraproject.org/en-US/fedora/f29/release-notes/developers/Development_Python/
Nice find!

If all non-EOL'd Fedora versions made that change, I think this fix would be trivial. Unfortunately, it looks like Fedora 28 doesn't have a definitive deprecation date yet and looking at the pattern in https://en.wikipedia.org/wiki/Fedora_version_history, it will probably be at least a few months.

With that in mind, I think we need to modify certbot-auto to work with both package names.

@ohemorange, do you have the interest/cycles to work on this?
I don't think I can get to it this sprint, but happy to do it in a future one.

Commit message: Add a step to handle python3 on fedora29

Patch to review:
diff --git a/letsencrypt-auto-source/letsencrypt-auto b/letsencrypt-auto-source/letsencrypt-auto
index ce3b35e8628..382c26fd3c0 100755
--- a/letsencrypt-auto-source/letsencrypt-auto
+++ b/letsencrypt-auto-source/letsencrypt-auto
@@ -488,11 +488,17 @@ BOOTSTRAP_RPM_PYTHON3_VERSION=1
 BootstrapRpmPython3() {
   # Tested with:
   #   - CentOS 6
+  #   - Fedora 29
 
   InitializeRPMCommonBase
 
+  # Fedora 29 must use python3-virtualenv
+  if $TOOL list python3-virtualenv >/dev/null 2>&1; then
+    python_pkgs="python3
+      python3-virtualenv
+    "
   # EPEL uses python34
-  if $TOOL list python34 >/dev/null 2>&1; then
+  elif $TOOL list python34 >/dev/null 2>&1; then
     python_pkgs="python34
       python34-devel
       python34-tools
@@ -741,7 +747,10 @@ elif [ -f /etc/redhat-release ]; then
   prev_le_python="$LE_PYTHON"
   unset LE_PYTHON
   DeterminePythonVersion "NOCRASH"
-  if [ "$PYVER" -eq 26 ]; then
+  # Starting to Fedora 29, python2 is on a deprecation path. Let's move to python3 then.
+  RPM_DIST_NAME=`(source /etc/os-release && echo $ID) || echo "unknown"`
+  RPM_DIST_VERSION=`(source /etc/os-release && echo $VERSION_ID) || echo "0"`
+  if [ "$RPM_DIST_NAME" = "fedora" -a "$RPM_DIST_VERSION" -ge 29 -o "$PYVER" -eq 26 ]; then
     Bootstrap() {
       BootstrapMessage "RedHat-based OSes that will use Python3"
       BootstrapRpmPython3
diff --git a/letsencrypt-auto-source/letsencrypt-auto.template b/letsencrypt-auto-source/letsencrypt-auto.template
index 2cd3f4336ec..1102855fccb 100755
--- a/letsencrypt-auto-source/letsencrypt-auto.template
+++ b/letsencrypt-auto-source/letsencrypt-auto.template
@@ -323,7 +323,10 @@ elif [ -f /etc/redhat-release ]; then
   prev_le_python="$LE_PYTHON"
   unset LE_PYTHON
   DeterminePythonVersion "NOCRASH"
-  if [ "$PYVER" -eq 26 ]; then
+  # Starting to Fedora 29, python2 is on a deprecation path. Let's move to python3 then.
+  RPM_DIST_NAME=`(source /etc/os-release && echo $ID) || echo "unknown"`
+  RPM_DIST_VERSION=`(source /etc/os-release && echo $VERSION_ID) || echo "0"`
+  if [ "$RPM_DIST_NAME" = "fedora" -a "$RPM_DIST_VERSION" -ge 29 -o "$PYVER" -eq 26 ]; then
     Bootstrap() {
       BootstrapMessage "RedHat-based OSes that will use Python3"
       BootstrapRpmPython3
diff --git a/letsencrypt-auto-source/pieces/bootstrappers/rpm_python3.sh b/letsencrypt-auto-source/pieces/bootstrappers/rpm_python3.sh
index b011a7235b2..545d45b46a2 100644
--- a/letsencrypt-auto-source/pieces/bootstrappers/rpm_python3.sh
+++ b/letsencrypt-auto-source/pieces/bootstrappers/rpm_python3.sh
@@ -5,11 +5,17 @@ BOOTSTRAP_RPM_PYTHON3_VERSION=1
 BootstrapRpmPython3() {
   # Tested with:
   #   - CentOS 6
+  #   - Fedora 29
 
   InitializeRPMCommonBase
 
+  # Fedora 29 must use python3-virtualenv
+  if $TOOL list python3-virtualenv >/dev/null 2>&1; then
+    python_pkgs="python3
+      python3-virtualenv
+    "
   # EPEL uses python34
-  if $TOOL list python34 >/dev/null 2>&1; then
+  elif $TOOL list python34 >/dev/null 2>&1; then
     python_pkgs="python34
       python34-devel
       python34-tools


Code review comments:
diff_hunk: @@ -323,7 +323,10 @@ elif [ -f /etc/redhat-release ]; then
   prev_le_python="$LE_PYTHON"
   unset LE_PYTHON
   DeterminePythonVersion "NOCRASH"
-  if [ "$PYVER" -eq 26 ]; then
+  # Starting to Fedora 29, python2 is on a deprecation path. Let's move to python3 then.
+  RPM_DIST_NAME=`(source /etc/os-release && echo $ID) || echo "unknown"`
+  RPM_DIST_VERSION=`(source /etc/os-release && echo $VERSION_ID) || echo "0"`
path: letsencrypt-auto-source/letsencrypt-auto.template
text: @user1:
```suggestion
  RPM_DIST_VERSION=`(. /etc/os-release && echo $VERSION_ID) || echo "0"`
```
`/bin/sh` seems to be bash on Fedora and I'm not sure if this will ever change, but `source` isn't part of the POSIX standard. [. is though](http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#dot) so I think we should use it just to avoid any problems there if Fedora or an individual user changes things in the future.

diff_hunk: @@ -323,7 +323,10 @@ elif [ -f /etc/redhat-release ]; then
   prev_le_python="$LE_PYTHON"
   unset LE_PYTHON
   DeterminePythonVersion "NOCRASH"
-  if [ "$PYVER" -eq 26 ]; then
+  # Starting to Fedora 29, python2 is on a deprecation path. Let's move to python3 then.
+  RPM_DIST_NAME=`(source /etc/os-release && echo $ID) || echo "unknown"`
path: letsencrypt-auto-source/letsencrypt-auto.template
text: @user1:
```suggestion
  RPM_DIST_NAME=`(. /etc/os-release && echo $ID) || echo "unknown"`
```
`/bin/sh` seems to be bash on Fedora and I'm not sure if this will ever change, but `source` isn't part of the POSIX standard. [. is though](http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#dot) so I think we should use it just to avoid any problems there if Fedora or an individual user changes things in the future.

@author:
Yes you are right.