Repository: certbot/certbot

Issue:
Certbot-auto 0.30.1 fails to install with pip 19.0.1
If you're having trouble using Certbot and aren't sure you've found a bug or
request for a new feature, please first try asking for help at
https://community.letsencrypt.org/. There is a much larger community there of
people familiar with the project who will be able to more quickly answer your
questions.

## My operating system is (include version):
Ubuntu 18.0.4.1 LTS

## I installed Certbot with (certbot-auto, OS package manager, pip, etc):
using certbot-auto via git repo (installed via `git clone https://github.com/certbot/certbot.git`) 

## I ran this command and it produced this output:
`sudo ./certbot-auto --no-bootstrap`
```
Creating virtual environment...
Installing Python packages...
Command "/opt/eff.org/certbot/venv/bin/python2.7 -u -c "import setuptools, tokenize;file='/tmp/pip-3yySnw-build/setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" install --record /tmp/pip-oamDAe-record/install-record.txt --single-version-externally-managed --compile --install-headers /opt/eff.org/certbot/venv/include/site/python2.7/wheel" failed with error code 1 in /tmp/pip-3yySnw-build/
Traceback (most recent call last):
File "/tmp/tmp.cfxY1NQimR/pipstrap.py", line 177, in 
exit(main())
File "/tmp/tmp.cfxY1NQimR/pipstrap.py", line 164, in main
shell=True)
File "/usr/lib/python2.7/subprocess.py", line 223, in check_output
raise CalledProcessError(retcode, cmd, output=output)
subprocess.CalledProcessError: Command 'pip install --no-index --no-deps -U --no-cache-dir /tmp/pipstrap-eTH3eH/pip-9.0.1.tar.gz /tmp/pipstrap-eTH3eH/setuptools-29.0.1.tar.gz /tmp/pipstrap-eTH3eH/wheel-0.29.0.tar.gz' returned non-zero exit status 1
```
## Certbot's behavior differed from what I expected because:
I expect the correct output:

```
Requesting to rerun ./certbot-auto with root privileges...
Creating virtual environment...
Installing Python packages...
Installation succeeded.
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator apache, Installer apache

Which names would you like to activate HTTPS for?
```
## Here is a Certbot log showing the issue (if available):
###### Logs are stored in `/var/log/letsencrypt` by default. Feel free to redact domains, e-mail and IP addresses as you see fit.
NA
## Here is the relevant nginx server block or Apache virtualhost for the domain I am configuring:
NA

## Additional relevant information: 
Originally hopped on issue #6682 but was asked to open a new issue. This first began today with the update to 0.30.1. I upgraded pip accordingly and now have 19.0.1 installed. To troubleshoot I ran the following series of commands, which worked. 
```
sudo rm certbot-auto
wget https://dl.eff.org/certbot-auto
export VIRTUALENV_NO_DOWNLOAD=1
chmod a+x certbot-auto
./certbot-auto --no-bootstrap --no-self-upgrade
```
This leads me to believe something is broken when upgrade to 0.30.1. I also have blown away venv via `sudo rm -rf /opt/eff.org/certbot/venv`


Discussion:
Thanks for reporting over here! You mentioned that it was working last night for you, right? That's surprising to me. Do you have the logs from that? Looking for what version of certbot-auto was running, and ideally what version of pip and virtualenv you had before you ran certbot-auto, and if possible also what version of virtualenv and pip certbot-auto installed.
@ohemorange I just got it to work now by removing 0.30.1 and replacing it with 0.30.0 - see the end of my issue for all commands ran to remove certbot-auto. I can give you logs though if you still need. Pip version was still 19.0.1 and venv was a fresh install from today. 


*** EDIT *** Including logs from latest working run using certbot-auto 0.30.0:

```
2019-01-24 16:51:49,487:DEBUG:certbot.main:certbot version: 0.30.0
2019-01-24 16:51:49,488:DEBUG:certbot.main:Arguments: ['--no-bootstrap', '--no-self-upgrade']
2019-01-24 16:51:49,488:DEBUG:certbot.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#apache,PluginEntryPoint#manual,PluginEntryPoint#nginx,PluginEntryPoint#null,PluginEntryPoint#standalone,PluginEntryPoint#webroot)
2019-01-24 16:51:49,520:DEBUG:certbot.log:Root logging level set at 20
2019-01-24 16:51:49,521:INFO:certbot.log:Saving debug log to /var/log/letsencrypt/letsencrypt.log
2019-01-24 16:51:49,543:DEBUG:certbot.plugins.selection:Requested authenticator None and installer None
2019-01-24 16:51:51,713:DEBUG:certbot_apache.configurator:Apache version is 2.4.29
2019-01-24 16:51:52,233:DEBUG:certbot.plugins.disco:No installation (PluginEntryPoint#nginx): Could not find a usable 'nginx' binary. Ensure nginx exists, the binary is executable, and your PATH is set correctly.
Traceback (most recent call last):
  File "/opt/eff.org/certbot/venv/local/lib/python2.7/site-packages/certbot/plugins/disco.py", line 132, in prepare
    self._initialized.prepare()
  File "/opt/eff.org/certbot/venv/local/lib/python2.7/site-packages/certbot_nginx/configurator.py", line 145, in prepare
    "Could not find a usable 'nginx' binary. Ensure nginx exists, "
NoInstallationError: Could not find a usable 'nginx' binary. Ensure nginx exists, the binary is executable, and your PATH is set correctly.
2019-01-24 16:51:52,236:DEBUG:certbot.plugins.selection:Single candidate plugin: * apache
Description: Apache Web Server plugin
Interfaces: IAuthenticator, IInstaller, IPlugin
Entry point: apache = certbot_apache.entrypoint:ENTRYPOINT
Initialized: <certbot_apache.override_debian.DebianConfigurator object at 0x7ff91112b490>
Prep: True
2019-01-24 16:51:52,238:DEBUG:certbot.plugins.selection:Selected authenticator <certbot_apache.override_debian.DebianConfigurator object at 0x7ff91112b490> and installer <certbot_apache.override_debian.DebianConfigurator object at 0x7ff91112b490>
2019-01-24 16:51:52,238:INFO:certbot.plugins.selection:Plugins selected: Authenticator apache, Installer apache
2019-01-24 16:51:52,283:DEBUG:certbot.main:Picked account: <Account(RegistrationResource(body=Registration(status=None, terms_of_service_agreed=None, agreement=None, only_return_existing=None, contact=(), key=None, external_account_binding=None), uri=u'https://acme-v02.api.letsencrypt.org/acme/acct/49044082', new_authzr_uri=None, terms_of_service=None), 74e36fcbd077f930cb48beddf5b4f520, Meta(creation_host=u'<redacted>', creation_dt=datetime.datetime(2019, 1, 6, 22, 48, 7, tzinfo=<UTC>)))>
2019-01-24 16:51:52,284:DEBUG:acme.client:Sending GET request to https://acme-v02.api.letsencrypt.org/directory.
2019-01-24 16:51:52,304:DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): acme-v02.api.letsencrypt.org:443
2019-01-24 16:51:53,021:DEBUG:urllib3.connectionpool:https://acme-v02.api.letsencrypt.org:443 "GET /directory HTTP/1.1" 200 658
2019-01-24 16:51:53,026:DEBUG:acme.client:Received response:
HTTP 200
Server: nginx
Content-Type: application/json
Content-Length: 658
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800
Expires: Fri, 25 Jan 2019 00:51:52 GMT
Cache-Control: max-age=0, no-cache, no-store
Pragma: no-cache
Date: Fri, 25 Jan 2019 00:51:52 GMT
Connection: keep-alive

{
  "keyChange": "https://acme-v02.api.letsencrypt.org/acme/key-change",
  "meta": {
    "caaIdentities": [
      "letsencrypt.org"
    ],
    "termsOfService": "https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf",
    "website": "https://letsencrypt.org"
  },
  "newAccount": "https://acme-v02.api.letsencrypt.org/acme/new-acct",
  "newNonce": "https://acme-v02.api.letsencrypt.org/acme/new-nonce",
  "newOrder": "https://acme-v02.api.letsencrypt.org/acme/new-order",
  "revokeCert": "https://acme-v02.api.letsencrypt.org/acme/revoke-cert",
  "scb93YxIhrU": "https://community.letsencrypt.org/t/adding-random-entries-to-the-directory/33417"
}
```
We're currently looking into this. In the meantime, the most correct fix for anyone else running into this is to use the ppa on Ubuntu 18.04. See instructions for Apache [here](https://certbot.eff.org/lets-encrypt/ubuntubionic-apache) and Nginx [here](https://certbot.eff.org/lets-encrypt/ubuntubionic-nginx).
Hey @erikrose -- looks like changing the pinned version of setuptools in pipstrap to 30.3.0 fixes this issue. How do you find the URL to get the tarball directly from?And is it just a hash of the tarball in there?

This might lead us into problems with `--no-deps` but the easiest way to figure that out is to test that.
@ohemorange Thanks for that! I did start to setup ppa for Ubuntu, however I wanted to use certbot-auto and I didn't think I could use that with the ppa. Am I incorrect?
Nice work finding that.

Not Erik but I've written/reviewed a few PRs to pipstrap over the years...

> How do you find the URL to get the tarball directly from?

If you look up the package and version on PyPI, you can find the link for the tarball. For instance, here's all the files for that version of setuptools. The full URL is:

https://pypi.python.org/packages/e0/b4/1e023e0f0e3a1475c146889f91989a306162c162d76eceac8ad13d6fad8d/setuptools-30.3.0.tar.gz

(The hostname shown by PyPI is different, but the path is the same. If you try and fetch the URL shown above, it redirects you a bunch before you wind up at the hostname given by PyPI.)

> And is it just a hash of the tarball in there?

Yep. SHA-256 hash of the file. The hash for the file linked above is 2ea217cc858de2ad59c87a5ccb4d22422abd8cf7235803b06df7983e1a581096.

> This might lead us into problems with --no-deps but the easiest way to figure that out is to test that.

Yeah. This will require a lot of testing and maybe also carefully looking at `setuptools` to make sure the changes they made between these versions won't cause any trouble for us.

Erik and I looked into this before the last time we bumped the version of setuptools used here. Going to dig a bit and see if I can find the relevant discussion and concerns.

EDIT: That discussion was at https://github.com/mozilla/pipstrap/pull/17. Looking through the changelog, we may be able to go as recent as 33.1.1.
@jldeen `certbot-auto` is just a wrapper around `certbot`, it does all of the same things, the `auto` just refers to the installation method that automatically updates immediately when we do a release. You might be referring to the automatic renewal functionality, which is present in `certbot`.
@ohemorange Yes, the automatic renewal functionality and walk through menu was what I was looking for. I could just use cron to auto renew, but the ease of certbot-auto was what led me down the wrapper path. Huge thanks to everyone for jumping on a fix for this issue, btw!

Resolving patch:
diff --git a/letsencrypt-auto-source/pieces/pipstrap.py b/letsencrypt-auto-source/pieces/pipstrap.py
--- a/letsencrypt-auto-source/pieces/pipstrap.py
+++ b/letsencrypt-auto-source/pieces/pipstrap.py
@@ -74,9 +74,9 @@ def check_output(*popenargs, **kwargs):
      'pip-{0}.tar.gz'.format(PIP_VERSION),
      '09f243e1a7b461f654c26a725fa373211bb7ff17a9300058b205c61658ca940d'),
     # This version of setuptools has only optional dependencies:
-    ('59/88/2f3990916931a5de6fa9706d6d75eb32ee8b78627bb2abaab7ed9e6d0622/'
-     'setuptools-29.0.1.tar.gz',
-     'b539118819a4857378398891fa5366e090690e46b3e41421a1e07d6e9fd8feb0'),
+    ('37/1b/b25507861991beeade31473868463dad0e58b1978c209de27384ae541b0b/'
+     'setuptools-40.6.3.zip',
+     '3b474dad69c49f0d2d86696b68105f3a6f195f7ab655af12ef9a9c326d2b08f8'),
     ('c9/1d/bd19e691fd4cfe908c76c429fe6e4436c9e83583c4414b54f6c85471954a/'
      'wheel-0.29.0.tar.gz',
      '1ebb8ad7e26b448e9caa4773d2357849bf80ff9e313964bcaf79cbf0201a1648')


Test patch: