Repository: certbot/certbot

Issue:
Install Python3 via EPEL on CentOS6 with letsencrypt-auto?
CentOS 6 ships with Python 2.6 by default, which isn't 100% stably supported by letsencrypt, and which the Python community at large is trying to drop support for. But there's good news! CentOS has support for Software Collections which make it possible to install Python 2.7 in parallel: https://www.softwarecollections.org/en/scls/rhscl/python27/

Would it make sense for `letsencrypt-auto` to just install a Python 2.7, in addition to all the other packages it provides, in order to a) ensure users always get a fully functional environment, b) help encourage folks to move to Python 2.7



Discussion:
@hogarthj what are your thoughts on this?

I got this need too (using the already installed scl repository and python27) and plan to made the necessaries changes. 
If you wanna to install scl and python during bootstrap, the determine python version function used in letsencrypt-auto will need another check to cover the specific python path

+1
Doing this allows us to remove Python 2.6 support which fixes or helps make progress on a lot of issues like #3726, #4244, #4377, and #4379. @SwartzCr put me in touch with some people from Red Hat who told us how to reliably and safely do this programmaticly. Here's what I learned:

OK, so the first thing to do would be to get a copy of Python 2.7 onto the system. RHEL and CentOS 6 both provide a technology called "Software Collections" that allow users to install a version of Python 2.7 (along with many other useful tools) outside of the usual system paths.

On CentOS, this is accomplished by installing the `centos-release-scl` package (`yum install centos-release-scl`) followed by `yum install python27`.

On RHEL, this may depend on which subscriptions are available; it may be necessary to add the "Optional" set of repositories to the RHEL subscription in order to install the python27 package.
```
subscription-manager repos --enable rhel-6-server-optional-rpms
subscription-manager repos --enable rhel-server-rhscl-6-rpms
```
On some RHEL systems, `rhn-channel` must be used instead of subscription manager. You can determine which program must be used by running:
```
yum -C repolist |grep 'receiving updates from RHN Classic'
```

or
```
yum -C repolist |grep 'receiving updates from Red Hat Subscription Management'
```
(Also you should check /etc/redhat-release first and make sure you're on RHEL vs. CentOS or Scientific Linux)

More info about how to do this can be found at: https://access.redhat.com/solutions/472163

After that, `yum install python27` should work.

Once those packages are installed, it will be possible to issue commands to this python27 installation by using the following syntax: scl enable python27 'command --arg1 --arg2'

Alternately, you can have your tools wrapped in a bash script that will `source /opt/rh/python27/enable` which will set your environment variables appropriately so that paths and other things are looking in the right places. This file is just environment variables, so you can probably parse it and set it in your script directly as well.

Additional resources:

* https://www.madboa.com/blog/2016/08/29/scl-intro/
* https://access.redhat.com/documentation/en-US/Red_Hat_Software_Collections/2-Beta/html/Packaging_Guide/sect-Enabling_the_Software_Collection.html

cc @joohoi, @ohemorange 

EDIT: We'll need to check `/etc/issue` or something to make sure we only do this on CentOS/RHEL 6. The current behavior of `certbot-auto` on CentOS/RHEL 7 (which we don't recommend people use because we're packaged) is fine and doesn't need to install/enable all this extra stuff.
Are we packaged on any of these systems?
And what do we do if we're on Scientific Linux, whatever that is?
> Are we packaged on any of these systems?

We're packaged in [EPEL](https://fedoraproject.org/wiki/EPEL) on CentOS/RHEL 7, but Python 2.7 is already available in the standard repositories. We're not packaged for CentOS/RHEL 6 which is the last platform we support using Python 2.6.

> And what do we do if we're on Scientific Linux, whatever that is?

I doubt anyone has tested `certbot-auto` on Scientific Linux, but what we should currently do is error out with:
```
Sorry, I don't know how to bootstrap Certbot on your operating system!

You will need to bootstrap, configure virtualenv, and run pip install manually.
Please see https://letsencrypt.readthedocs.org/en/latest/contributing.html#prerequisites
for more info.
```
AFAIK, no one has ever complained about this and I think this is fine behavior for now.
I didn't realize RHEL was the last holdout. Let's do it! (RHEL people, please tell me how unreasonable I'm being.)
I don't think you're being unreasonable at all! My instructions above come from Red Hat employees who I asked how we can still support CentOS/RHEL 6 without Python 2.6 support. Happy to talk with you or them about this more if you have questions.
One complication here is getting SCL and Python 2.7 installed for existing `certbot-auto` users on these systems.
It may be time to teach certbot-auto how to decide in a more fine-grained fashion whether the OS packages are sufficiently bootstrapped. :-)
I agree. Any ideas?
@erikrose, since I'm planning on working on this for the next release, I'd love to see your suggestions for how to handle the problem of new OS dependencies in `certbot-auto` when you get a chance.
I'm not going to have much time until after PyCon, so I'll just give you a quick sketch:

1. Keep track of a monotonically increasing version number of OS deps for each machine. Write DB-style "migrations" to move from each version to the next, installing whatever new deps are necessary. At install time, run as many of those migrations as needed to get to the latest version. The downside of this is that, if the user manually removes a dep in the interrim, we never recover.

2. Bullheadedly try to install all the OS deps each time. I believe we quit doing this because it was network-intense and time-consuming, but I could stand to have my memory refreshed. Maybe we could write a more parsimonious version that locally queries what packages are installed (in an OS-specific way) and compares against that. Writing all those comparators sounds like no fun to me.
@bmw sorry I missed this notification in a flood of other things

Using SCL for this would be the best supported method, and would keep to CentOS built packages and RHEL built packages.

I can't use SCL for EPEL unfortunately ... the best thing would be python27 from SCL and then a virtualenv pip on EL6 right now.


So, I went to look at the data from PyPI, and it appears that CentOS absolutely _dwarfs_ RHEL for number of CertBot installs with Python 2.6, so a solution that only worked on CentOS (which sounds like it'd be much simpler) seems like it'd get us 90%+ of the win.

Here's the result of the query:

```sql
SELECT
  details.distro.name,
  details.distro.version,
  COUNT(*) as download_count,
FROM
  TABLE_DATE_RANGE(
    [the-psf:pypi.downloads],
    DATE_ADD(CURRENT_TIMESTAMP(), -4, "week"),
    CURRENT_TIMESTAMP()
  )
WHERE
  REGEXP_EXTRACT(details.python, r"^([^\.]+\.[^\.]+)") = "2.6" AND file.project = 'certbot'
GROUP BY
  details.distro.name,
  details.distro.version
ORDER BY
  download_count DESC
LIMIT 100
```
![screen shot 2017-05-23 at 8 33 45 am](https://cloud.githubusercontent.com/assets/772/26362373/90813a44-3f92-11e7-80e1-c2dba0404a1c.png)


@alex That's really interesting.  How do those compare with RHEL/CentOS 7 numbers?
Didn't look, this query was for "which distros cause the most 2.6 downloads".
We see similar numbers on our side. The data below was collected recently for two months of certificate issuances for Certbot where "_count" is the number of certs issued:
```
"os","_count"
"CentOS Linux 7 (Core)","416383"
"CentOS 6.5","188968"
"CentOS 6.8","136343"
"CentOS 6.6","67779"
"CentOS 6.4","54571"
"CentOS 6.9","34327"
"CentOS 6.7","22848"
"CentOS 6.3","2462"
"Red Hat Enterprise Linux Server 7.3 (Maipo)","1854"
"Red Hat Enterprise Linux Server 6.8","1224"
"Red Hat Enterprise Linux Server 6.9","1185"
"CentOS Linux 7.3.1611","1116"
"Red Hat Enterprise Linux","831"
"CentOS 6.2","794"
"CentOS Linux 7.2.1511","785"
"CentOS Linux 6.0","513"
"Red Hat Enterprise Linux Server 7.2 (Maipo)","506"
```
We could potentially do something for just CentOS first if:
1. We had `certbot-auto` keep them on the latest version that works rather than trying to upgrade and breaking.
2. We added support for RHEL in the future so they eventually saw updates.
Rather than enabling the additional SCL repos, I'm considering just installing Python 3 from EPEL which we already require to be enabled.
Sounds like a pretty good idea:
* People hate it when we install more repos.
* We're already testing against Python 3.
* It gives us a staged rollout of Python 3 as a side effect, which is better than the big bang we otherwise would eventually need.
Proposal:

* Add a version number to each bootstrap script that must be incremented when new OS dependencies are added.
* Write a function in `certbot-auto` that returns the current dependency version string for the OS.
* Add a `--os-version` flag to `certbot-auto` that has it print this version string and exit. After downloading a new `certbot-auto` but before replacing the currently running script, execute something like the following bashy pseudocode:
```
MY_OS_VERSION=$(GetOSVersion)
NEW_OS_VERSION=$("$TEMP_DIR/letsencrypt-auto.permission-clone" --os-version)
if $MY_OS_VERSION != $NEW_OS_VERSION; then
  # did the user specify --non-interactive or are stdin and stdout attached to a terminal?
  if $ASSUME_YES = 1 || (test -t 0 && test -t 1); then
    # delete the venv so certbot-auto rebootstraps
    rm -rf "$VENV_PATH"
    $SUDO mv -f "$TEMP_DIR/letsencrypt-auto.permission-clone" "$0"
    "$0" "$@"
  else
    echo Skipping upgrade because new OS dependencies must be installed and we want your approval.
    "$0" --le-auto-phase2 "$@"
  fi
else
  $SUDO mv -f "$TEMP_DIR/letsencrypt-auto.permission-clone" "$0"
  "$0" --le-auto-phase2 "$@"
fi
```
I kind of hate the terminal checks. Alternatively, we could prompt the user asking if they want to upgrade knowing they must install additional OS dependencies, but we'd also have to implement a timeout and this approach is significantly more work for minimal gain. If anyone has any ideas here they like better, I'd love to hear them.

We also currently treat the usage of `renew` on the command line the same as `--non-interactive`, but I'm going to remove that. Otherwise, people who throw `certbot-auto renew` in a cron job without `--no-self-upgrade` just automatically get whatever OS dependencies we want to install on their system. `certbot-auto` not asking for permission to do these things pissed people off before so let's not do it again.

EDIT: Added comments to confusing script.
EDIT2: Properly handle `--le-auto-phase2`. I'd do this cleaner in the real script, but this is the effect I had in mind.
You used a couple different terms throughout your comment. I'm going to go with "bootstrap version" because it doesn't sound like it's the 5 in "RHEL 5". I suggest we go with `--bootstrap-version` as well.

Currently, if the user manually uninstalls some dependency, they're out of luck: we'll never automatically recover. With this change, we *will* recover when next we increment the bootstrap version. So that's a nice side effect.

Removing the venv sounded extreme to me at first, but then I thought about those compiled Python libs that OpenSSL broke in an ill-considered update. Now I'm onboard.

~~We may want to lay down the bootstrap version as a little file in the venv and compare against that instead of against the number embedded in certbot-auto. I could imagine myself inheriting a machine on which there were several different versions of certbot-auto, unsure which one was the last run. certbot-auto doesn't have a standard place on the FS, but the venv does, making that a nice place to park machine-wide state.~~ Never mind; the venv location is based on which user runs the script.

Ah, but wait! Once upon a time, there was a machine upon which someone ran certbot-auto. It had bootstrap version 4. It was very proud of its bootstrap version and polished it every day. One day, a wicked witch bought a cert from Verisign and deleted the copy of certbot-auto. Later, the witch tripped over a bucket and fell down some stairs and was replaced by a cost-conscious dwarf who downloaded a fresh copy of certbot-auto (now bootstrap version 5) and ran it. The new certbot-auto was not doing a self-upgrade, so it neglected to do the compare and run the run the new bootstrapping script. If only we had stored the bootstrap version in the venv! (If it finds it, great—it does the right thing. If it's starting a new venv, also great—it always bootstraps.) We also should make sure all our bootstrapping scripts are idempotent, which I expect they are.
And also (discussed on IRC), I think we should add the bootstrap script name to the bootstrap version number. This is to defend us against machines moving across bootstrap-version namespaces. It shouldn't happen now, but I can envision a future in which we have a separate script for FooLinux 3 and FooLinux 8. `!=` will still do its job.
Saving the bootstrap version to disk and including the bootstrap script name are great ideas. As a minor point, since the built version of `certbot-auto` doesn't have a great way to tell where different pieces of code come from, I think we should include the name of the bootstrap function instead but I think this has the same effect. With this and our IRC discussion in mind, I propose the updated plan:

* Add a version number to each bootstrap script that must be incremented when new OS dependencies are added. This will be something like `DEB_BOOTSTRAP_VERSION`.
* Take the giant `if [ -f /etc/debian_version ]` logic out of `Bootstrap()`. Instead, this `if` `else` tree will execute on every run and set two variables. First, it will set `Bootstrap` to the function to run to install OS dependencies on this system. Second, it will set `BOOTSTRAP_VERSION` to `"<bootstrap function name> <bootstrap version number>"`. If the user set `--no-bootstrap`, `Bootstrap()` will be a noop and `BOOTSTRAP_VERSION` will be a dummy value.
* Write a function that returns the bootstrap version stored on disk. If the file doesn't exist, it will return something like "Unknown 0".
* When creating the virtualenv, write a file containing the current bootstrap version named something like "certbot-bootstrap-version.txt". The assumption here is when coupled with the code below, if we're creating the venv, we just bootstrapped (or we were ran with `--no-bootstrap`).
* Update the beginning of [phase 2](https://github.com/certbot/certbot/blob/98805ccae01205537cbfa48dfa3704982fc128dc/certbot-auto#L658) to look something like:
```
# If the venv doesn't exist and we're in phase 2, we just bootstrapped so don't do it again
if [ -d "$VENV_PATH" -a "$BOOTSTRAP_VERSION" != $(SavedBootstrapVersion) ]; then
  # noninteractive mode or stdin and stdout are connected to a terminal
  if [ $ASSUME_YES = 1 -o -t 0 -a -t 1]; then
    rm -rf "$VENV_PATH"
    exec "$0" "$@"
  else
    echo Skipping upgrade because new OS dependencies need to be installed.
    # Set INSTALLED_VERSION to be the same so we don't update the venv
    INSTALLED_VERSION=$LE_AUTO_VERSION
  fi
elif [ -f "$VENV_BIN/letsencrypt" ]; then
...
```
Seems sane on the face of it. Will look harder once the code takes shape.
I have the mechanism for doing the upgrades mostly implemented in the `os-auto-update-system` branch. Making use of it to stop using Python 2.6 will come in a later PR.

I'm currently a little stuck on the following problem though:

I suspect most users who have gotten `certbot-auto` to run automatically from cron or a systemd service are running the script as root. This is because otherwise the script will try to run `sudo`/`su` which doesn't work (by default) in cron or systemd services. This means that Certbot is installed at `/root/.local/share/letsencrypt`.

Additionally, I suspect most users don't use `sudo`/`su` themselves when they run the script manually. Until a very recent change to our website, we never told them to because `certbot-auto` runs `sudo`/`su` as necessary internally. (The fact that we now tell users to run `sudo certbot-auto` is an unattended side-effect of another change.) This (coupled with the fact that `sudo` preserves `$HOME` by default on many systems) means that Certbot is installed at `~/.local/share/letsencrypt` where `~` is not `/root`.

Because of this, I suspect most `certbot-auto` users who are using cron/systemd have two Certbot installations. Why does this matter here? Well when you start getting root e-mail from cron/systemd saying that Certbot has an upgrade available but needs your attention, you'll log in and run `certbot-auto` and upgrade, but the `root` installation of `certbot-auto` has no idea that this has happened and continues to try and get your attention.

Potential mitigations I've thought of are:

1. Put `certbot-auto-bootstrap-version.txt` in one of Certbot's configuration directories, but this is much more visible to the user and breaks if the user changes directories or deletes their configuration to start over.
2. Change the default installation location of `certbot-auto` to work around this problem. We can do this, but it's non-trivial code to write to ensure upgrades continue to work and will break users using `venv/bin` directly (we know there are at least a few).
3. Try and tell the sysadmin which user to run `certbot-auto` as. I haven't come up with a reliable way to do this.
I'm for (1) (though /var/lib is probably a more appropriate spot than /etc, according to the Filesystem Hierarchy Standard). Advantages:

* Doesn't require the sysadmin to read or know any additional things
* Puts machine-state stuff in a machine-global place
* Doesn't require certbot-auto to have a One True Location, which would be kind of a weird thing to require when it's a self-updating whatzit that you download and put where you want. (Programs that move when you open them make me…uneasy.)
@bmw Is the issue you described with multiple certbot installs specific to this, or would any upgrade for certbot trigger it?
It's specific to the issue of us adding new dependencies we can't install in the virtual environment ourselves through `pip`. That issue was resolved with #4970 though.

To bring this issue up to date, here's what still needs to be done:

1. We need to set `BOOTSTRAP_VERSION` [here](https://github.com/certbot/certbot/blob/5f6b1378eca5a157a35f36e778e8b26702d758d0/certbot-auto#L717) to a different value only for those systems we're going to force to Python 3. Changing this value will cause `certbot-auto` to (re)invoke the user's package manager or print a message that we need the user to run `certbot-auto` interactively to upgrade so we only want to change this value on the systems we have to.
2. In the same place we set `BOOTSTRAP_VERSION`, we need to set `Bootstrap` to a function that can be run to install OS dependencies on CentOS/RHEL 6 including Python 3. This should probably be done by creating a new [bootstrap script](https://github.com/certbot/certbot/tree/5f6b1378eca5a157a35f36e778e8b26702d758d0/letsencrypt-auto-source/pieces/bootstrappers) and refactor `rpm_common.sh` in that directory to allow this script to reuse common code.
3. Ensure Python 3 is used when creating the virtualenv. This should be doable by setting the environment variable `LE_PYTHON` to the name of the Python 3 executable in the user's PATH and exporting it.
4. Profit?
@hlieberman This was the issue I mentioned tonight.

Resolving patch:
diff --git a/letsencrypt-auto-source/pieces/fetch.py b/letsencrypt-auto-source/pieces/fetch.py
--- a/letsencrypt-auto-source/pieces/fetch.py
+++ b/letsencrypt-auto-source/pieces/fetch.py
@@ -11,17 +11,22 @@
 
 """
 
-from __future__ import print_function
+from __future__ import print_function, unicode_literals
 
 from distutils.version import LooseVersion
 from json import loads
 from os import devnull, environ
 from os.path import dirname, join
 import re
+import ssl
 from subprocess import check_call, CalledProcessError
 from sys import argv, exit
-from urllib2 import build_opener, HTTPHandler, HTTPSHandler
-from urllib2 import HTTPError, URLError
+try:
+    from urllib2 import build_opener, HTTPHandler, HTTPSHandler
+    from urllib2 import HTTPError, URLError
+except ImportError:
+    from urllib.request import build_opener, HTTPHandler, HTTPSHandler
+    from urllib.error import HTTPError, URLError
 
 PUBLIC_KEY = environ.get('LE_AUTO_PUBLIC_KEY', """-----BEGIN PUBLIC KEY-----
 MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA6MR8W/galdxnpGqBsYbq
@@ -43,8 +48,11 @@ class HttpsGetter(object):
     def __init__(self):
         """Build an HTTPS opener."""
         # Based on pip 1.4.1's URLOpener
-        # This verifies certs on only Python >=2.7.9.
-        self._opener = build_opener(HTTPSHandler())
+        # This verifies certs on only Python >=2.7.9, and when NO_CERT_VERIFY isn't set.
+        if environ.get('NO_CERT_VERIFY') == '1' and hasattr(ssl, 'SSLContext'):
+            self._opener = build_opener(HTTPSHandler(context=create_CERT_NONE_context()))
+        else:
+            self._opener = build_opener(HTTPSHandler())
         # Strip out HTTPHandler to prevent MITM spoof:
         for handler in self._opener.handlers:
             if isinstance(handler, HTTPHandler):
@@ -66,7 +74,7 @@ def get(self, url):
 
 def write(contents, dir, filename):
     """Write something to a file in a certain directory."""
-    with open(join(dir, filename), 'w') as file:
+    with open(join(dir, filename), 'wb') as file:
         file.write(contents)
 
 
@@ -74,13 +82,13 @@ def latest_stable_version(get):
     """Return the latest stable release of letsencrypt."""
     metadata = loads(get(
         environ.get('LE_AUTO_JSON_URL',
-                    'https://pypi.python.org/pypi/certbot/json')))
+                    'https://pypi.python.org/pypi/certbot/json')).decode('UTF-8'))
     # metadata['info']['version'] actually returns the latest of any kind of
     # release release, contrary to https://wiki.python.org/moin/PyPIJSON.
     # The regex is a sufficient regex for picking out prereleases for most
     # packages, LE included.
     return str(max(LooseVersion(r) for r
-                   in metadata['releases'].iterkeys()
+                   in iter(metadata['releases'].keys())
                    if re.match('^[0-9.]+$', r)))
 
 
@@ -97,7 +105,7 @@ def verified_new_le_auto(get, tag, temp_dir):
         'letsencrypt-auto-source/') % tag
     write(get(le_auto_dir + 'letsencrypt-auto'), temp_dir, 'letsencrypt-auto')
     write(get(le_auto_dir + 'letsencrypt-auto.sig'), temp_dir, 'letsencrypt-auto.sig')
-    write(PUBLIC_KEY, temp_dir, 'public_key.pem')
+    write(PUBLIC_KEY.encode('UTF-8'), temp_dir, 'public_key.pem')
     try:
         with open(devnull, 'w') as dev_null:
             check_call(['openssl', 'dgst', '-sha256', '-verify',
@@ -112,6 +120,14 @@ def verified_new_le_auto(get, tag, temp_dir):
                             "certbot-auto.", exc)
 
 
+def create_CERT_NONE_context():
+    """Create a SSLContext object to not check hostname."""
+    # PROTOCOL_TLS isn't available before 2.7.13 but this code is for 2.7.9+, so use this.
+    context = ssl.SSLContext(ssl.PROTOCOL_SSLv23)
+    context.verify_mode = ssl.CERT_NONE
+    return context
+
+
 def main():
     get = HttpsGetter().get
     flag = argv[1]


Test patch:
diff --git a/letsencrypt-auto-source/tests/auto_test.py b/letsencrypt-auto-source/tests/auto_test.py
--- a/letsencrypt-auto-source/tests/auto_test.py
+++ b/letsencrypt-auto-source/tests/auto_test.py
@@ -202,6 +202,7 @@ def run_le_auto(le_auto_path, venv_dir, base_url, **kwargs):
 Z5o/NDiQB11m91yNB0MmPYY9QSbnOA9j7IaaC97AwRLuwXY+/R2ablTcxurWou68
 iQIDAQAB
 -----END PUBLIC KEY-----""",
+             NO_CERT_VERIFY='1',
              **kwargs)
     env.update(d)
     return out_and_err(
@@ -349,6 +350,7 @@ def test_openssl_failure(self):
                     self.assertTrue("Couldn't verify signature of downloaded "
                                     "certbot-auto." in exc.output)
                 else:
+                    print(out)
                     self.fail('Signature check on certbot-auto erroneously passed.')
 
     def test_pip_failure(self):
diff --git a/letsencrypt-auto-source/tests/centos6_tests.sh b/letsencrypt-auto-source/tests/centos6_tests.sh
new file mode 100644
--- /dev/null
+++ b/letsencrypt-auto-source/tests/centos6_tests.sh
@@ -0,0 +1,65 @@
+#!/bin/bash
+# Start by making sure your system is up-to-date:
+yum update > /dev/null
+yum install -y centos-release-scl > /dev/null
+yum install -y python27 > /dev/null 2> /dev/null
+
+# we're going to modify env variables, so do this in a subshell
+(
+source /opt/rh/python27/enable
+
+# ensure python 3 isn't installed
+python3 --version 2> /dev/null
+RESULT=$?
+if [ $RESULT -eq 0 ]; then
+  error "Python3 is already installed."
+  exit 1
+fi
+
+# ensure python2.7 is available
+python2.7 --version 2> /dev/null
+RESULT=$?
+if [ $RESULT -ne 0 ]; then
+  error "Python3 is not available."
+  exit 1
+fi
+
+# bootstrap, but don't install python 3.
+certbot/letsencrypt-auto-source/letsencrypt-auto --no-self-upgrade -n > /dev/null 2> /dev/null
+
+# ensure python 3 isn't installed
+python3 --version 2> /dev/null
+RESULT=$?
+if [ $RESULT -eq 0 ]; then
+  error "letsencrypt-auto installed Python3 even though Python2.7 is present."
+  exit 1
+fi
+
+echo ""
+echo "PASSED: Did not upgrade to Python3 when Python2.7 is present."
+)
+
+# ensure python2.7 isn't available
+python2.7 --version 2> /dev/null
+RESULT=$?
+if [ $RESULT -eq 0 ]; then
+  error "Python2.7 is still available."
+  exit 1
+fi
+
+# bootstrap, this time installing python3
+certbot/letsencrypt-auto-source/letsencrypt-auto --no-self-upgrade -n > /dev/null 2> /dev/null
+
+# ensure python 3 is installed
+python3 --version > /dev/null
+RESULT=$?
+if [ $RESULT -ne 0 ]; then
+  error "letsencrypt-auto failed to install Python3 when only Python2.6 is present."
+  exit 1
+fi
+
+echo "PASSED: Successfully upgraded to Python3 when only Python2.6 is present."
+echo ""
+
+# test using python3
+pytest -v -s certbot/letsencrypt-auto-source/tests