Repository: certbot/certbot

Issue:
Fix "oldest" tests on point release branches
In https://github.com/certbot/certbot/pull/6921, we made a change to properly update `local-oldest-requirements.txt` for point release branches, but it broke the tests. See https://travis-ci.com/certbot/certbot/jobs/197027330.

Until now, we've never had the case in the "oldest" tests where the pinned version like `certbot==0.34.0` matches the local files. Before either an older, non-local version was pinned or it was something like `-e acme` which instructs pip to install the local package into the virtual environment.

When we go to install the package in the oldest tests in the 0.34.x branch, we ask for `certbot==0.34.0`, which is found to be already available in the current working directory so pip does nothing.

Then when we change directories to run tests to prevent local files from impacting tests (which has been a problem in the past), the local package that was found before is no longer there, can't be imported, and the tests fail.
certbot-auto produces a stray newline in the standard output
Running certbot-auto on Debian Jessie, since the automatic upgrade to 0.34.0 each calls to certbot-auto outputs a new line

```
/usr/local/sbin/certbot-auto -q renew | hexdump -C
00000000  0a                                                |.|
00000001
```

The problem is that this in turn generates an empty cronjob mail. This was not happening with previous versions.

Thanks




Discussion:

Thanks for reporting this. We'll be putting out a point release to fix the problem, but you can work around it by adding `--no-permissions-check` to the command line in the meantime.
@bmw thanks!

Resolving patch:
diff --git a/tools/pip_install.py b/tools/pip_install.py
--- a/tools/pip_install.py
+++ b/tools/pip_install.py
@@ -99,8 +99,15 @@ def main(args):
         else:
             # Otherwise, we merge requirements to build the constraints and pin dependencies
             requirements = None
+            reinstall = False
             if os.environ.get('CERTBOT_OLDEST') == '1':
                 requirements = certbot_oldest_processing(tools_path, args, test_constraints)
+                # We need to --force-reinstall the tested distribution when using oldest
+                # requirements because of an error in these tests in particular situations
+                # described in https://github.com/certbot/certbot/issues/7014.
+                # However this slows down considerably the oldest tests (5 min -> 10 min),
+                # so we need to find a better mitigation in the future.
+                reinstall = True
             else:
                 certbot_normal_processing(tools_path, test_constraints)
 
@@ -109,8 +116,8 @@ def main(args):
                 pip_install_with_print('--constraint "{0}" --requirement "{1}"'
                                        .format(all_constraints, requirements))
 
-            pip_install_with_print('--constraint "{0}" {1}'
-                                   .format(all_constraints, ' '.join(args)))
+            pip_install_with_print('--constraint "{0}" {1} {2}'.format(
+                all_constraints, '--force-reinstall' if reinstall else '', ' '.join(args)))
     finally:
         if os.environ.get('TRAVIS'):
             print('travis_fold:end:install_certbot_deps')


Test patch:
diff --git a/tests/letstest/scripts/test_leauto_upgrades.sh b/tests/letstest/scripts/test_leauto_upgrades.sh
--- a/tests/letstest/scripts/test_leauto_upgrades.sh
+++ b/tests/letstest/scripts/test_leauto_upgrades.sh
@@ -15,8 +15,8 @@ if ! command -v git ; then
         exit 1
     fi
 fi
-# 0.18.0 is the oldest version of letsencrypt-auto that works on Fedora 26+.
-INITIAL_VERSION="0.18.0"
+# 0.33.x is the oldest version of letsencrypt-auto that works on Fedora 29+.
+INITIAL_VERSION="0.33.1"
 git checkout -f "v$INITIAL_VERSION" letsencrypt-auto
 if ! ./letsencrypt-auto -v --debug --version --no-self-upgrade 2>&1 | tail -n1 | grep "^certbot $INITIAL_VERSION$" ; then
     echo initial installation appeared to fail
diff --git a/tests/letstest/scripts/test_letsencrypt_auto_certonly_standalone.sh b/tests/letstest/scripts/test_letsencrypt_auto_certonly_standalone.sh
--- a/tests/letstest/scripts/test_letsencrypt_auto_certonly_standalone.sh
+++ b/tests/letstest/scripts/test_letsencrypt_auto_certonly_standalone.sh
@@ -42,8 +42,8 @@ if ! letsencrypt-auto --help --no-self-upgrade | grep -F "letsencrypt-auto [SUBC
     exit 1
 fi
 
-OUTPUT=$(letsencrypt-auto --install-only --no-self-upgrade --quiet 2>&1)
-if [ -n "$OUTPUT" ]; then
+OUTPUT_LEN=$(letsencrypt-auto --install-only --no-self-upgrade --quiet 2>&1 | wc -c)
+if [ "$OUTPUT_LEN" != 0 ]; then
     echo letsencrypt-auto produced unexpected output!
     exit 1
 fi