Repository: certbot/certbot

Issue:
Lexicon update causes urls to be malformed when adding TXT records
## My operating system is (include version):
Linux (amazon aws fargate v 1.3.0 container)

## I installed Certbot with (certbot-auto, OS package manager, pip, etc):
I am depending on the `certbot/dns-dnsimple` docker image in my Dockerfile.

## I ran this command and it produced this output:
When attempting to generate certificates for qa environments in my startup script (full package here: https://github.com/askwonder/wonqa) I run `docker run -i --name wonqa-certbot wonqa-certbot:latest certonly --dns-dnsimple --dns-dnsimple-credentials dnsimple.ini --agree-tos --no-eff-email -m eng@askwonder.com -d fix-qa.wondev.net,api.fix-qa.wondev.net,start.fix-qa.wondev.net,research.fix-qa.wondev.net,storybook.fix-qa.wondev.net,pusher.fix-qa.wondev.net`.
I get the following output: 

```
Obtaining a new certificate

Performing the following challenges:

dns-01 challenge for api.fix-qa.wondev.net
dns-01 challenge for fix-qa.wondev.net
dns-01 challenge for pusher.fix-qa.wondev.net
dns-01 challenge for research.fix-qa.wondev.net

dns-01 challenge for start.fix-qa.wondev.net
dns-01 challenge for storybook.fix-qa.wondev.net
Unsafe permissions on credentials configuration file: dnsimple.ini

Cleaning up challenges

Error adding TXT record: 400 Client Error: Bad Request for url: https://api.dnsimple.com/v281695/zones/wondev.net/records
```

## Certbot's behavior differed from what I expected because:
I expect it to add the TXT records successfully. Instead the url is malformed and there is no slash in between `v2` and `81695` which is my account id.

## Investigation
After doing some digging I have determined that the issue was caused by a change made by lexicon (https://github.com/AnalogJ/lexicon), likely this commit: https://github.com/AnalogJ/lexicon/commit/f46a072fc119d069bc276f7fd1977ab548fcd364#diff-89f3c260164ceef017a121144c790f42. 
I do not have a good explanation as to why we only started seeing issues due to this last week, or why that change appears to affect certbot despite you guys having locked your lexicon version at `dns-lexicon>=2.2.1` in your certbot-dns-dnsimple setup.py, but it has become a blocking issue for the wonqa repo referenced above. Fortunately, lexicon responded to our issue and merged a fix: https://github.com/AnalogJ/lexicon/pull/389.

Certbot can obtain that fix by upgrading your lexicon dependency. I have submitted a PR (https://github.com/certbot/certbot/pull/6928) and have tested that the proposed change does indeed resolve our issue. Obviously there may be additional considerations in bumping this dependency that I am not in a great position to test fully. But do let me know if there's any additional useful information that I can provide in the interim.





Discussion:
There are some complications in just updating the dependency used there, however, I just pushed the `latest` and `v0.33.1` tags to Docker Hub for the Certbot and dns-dnsimple images containing the fix for this.
Awesome, I'll give that a shot and let you know how it goes. Appreciate the quick response and action!
You can also merge our master branch to fix some other Travis builds in the mean time. 
Confirmed that @bmw's commit has resolved the issue for me on docker
Glad to hear this—I'm going to close this for now because I think the problem has been understood and fixed upstream, so eventually most channels and installation methods should have it.
Actually, I'm going to reopen this temporarily and mark it as "has PR", because I'm not certain what @adferrand and @bmw wanted to do with your PR.
Yeah I think leaving this open for now makes sense. We should do a little more here, although exactly what we should do in my opinion depends on what different distros do in response to this bug. See https://github.com/certbot/certbot/pull/6928#issuecomment-481020892 for a little more info.
To hijack this issue a bit, Debian and Ubuntu are planning on backporting fixes for this issue to Lexicon. This means that the certbot-dns-dnsimple plugin will continue to work in those distros without any effort on our part.

To ensure we're still able to ship updates to the certbot-dns-dnsimple plugin in those distros though, I think we want to test against the old Lexicon APIs to ensure we maintain compatibility with them. This is a bit awkward because we won't be able to successfully run integration tests on the dnsimple plugin with old versions of Lexicon, but these tests currently don't exist and they may never with some of the changes we're thinking about in the future to our DNS plugins (see https://github.com/certbot/certbot/issues/6504 or https://github.com/certbot/certbot/issues/6464). I personally think we can revisit this problem if it ever comes up.

With that said, I also think we should update the plugin's `setup.py` file to depend on `dns-lexicon>=3.2.1` with a comment explaining why this is needed. This ensures the plugin works by default and distros who have backported fixes can knowingly modify this requirement during packaging. Unfortunately, this complicates testing the plugin against old APIs because `setuptools` will complain that the plugin's dependencies aren't satisfied if an old version of Lexicon is installed.

With these constraints in mind, my proposed fix is to update `setup.py` to contain code like:
```
if os.environ.get('CERTBOT_OLDEST') == '1':
    install_requires.append('dns-lexicon>=2.2.1')
else:
    install_requires.append('dns-lexicon>=3.2.1')
fi
```
To ensure things like wheels are never incorrectly build depending on this version of Lexicon, I'd also want us to add `unset CERTBOT_OLDEST` to the release script before wheels are built.

What do you think @adferrand? This is certainly a little unconventional, but I think it pretty easily gets us the behavior we want.
Hi guys,

I'm running into this issue on Ubuntu 14.04. Is there a preferred hotfix right now? My domains are up for renewal in the next 14 days, may need to do it by hand if not...
I've pinged our PPA maintainers about this, but in the meantime, you can [use Docker](https://certbot.eff.org/docs/install.html#running-with-docker).

Resolving patch:
diff --git a/certbot-dns-dnsimple/setup.py b/certbot-dns-dnsimple/setup.py
--- a/certbot-dns-dnsimple/setup.py
+++ b/certbot-dns-dnsimple/setup.py
@@ -1,3 +1,4 @@
+import os
 from setuptools import setup
 from setuptools import find_packages
 
@@ -9,12 +10,23 @@
 install_requires = [
     'acme>=0.31.0',
     'certbot>=0.34.0.dev0',
-    'dns-lexicon>=2.2.1',  # Support for >1 TXT record per name
     'mock',
     'setuptools',
     'zope.interface',
 ]
 
+# This package normally depends on dns-lexicon>=3.2.1 to address the
+# problem described in https://github.com/AnalogJ/lexicon/issues/387,
+# however, the fix there has been backported to older versions of
+# lexicon found in various Linux distros. This conditional helps us test
+# that we've maintained compatibility with these versions of lexicon
+# which allows us to potentially upgrade our packages in these distros
+# as necessary.
+if os.environ.get('CERTBOT_OLDEST') == '1':
+    install_requires.append('dns-lexicon>=2.2.1')
+else:
+    install_requires.append('dns-lexicon>=3.2.1')
+
 docs_extras = [
     'Sphinx>=1.0',  # autodoc_member_order = 'bysource', autodoc_default_flags
     'sphinx_rtd_theme',


Test patch: