Repository: certbot/certbot

Issue:
Upgrade to dns-digitalocean 1.6 broke renewals
certbot/dns-digitalocean:v1.6.0

```
$ certbot renew
...
Processing /etc/letsencrypt/renewal/omcr.io.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Cert is due for renewal, auto-renewing...
Plugins selected: Authenticator dns-digitalocean, Installer None
Renewing an existing certificate
Performing the following challenges:
dns-01 challenge for omcr.io
Waiting 10 seconds for DNS changes to propagate
Waiting for verification...
Challenge failed for domain omcr.io
dns-01 challenge for omcr.io
Cleaning up challenges
Attempting to renew cert (omcr.io) from /etc/letsencrypt/renewal/omcr.io.conf produced an unexpected error: Some challenges have failed.. Skipping.
...
 - The following errors were reported by the server:

   Domain: omcr.io
   Type:   unauthorized
   Detail: Incorrect TXT record
   "mKIy_DFn65L85GgEMgl8BYiKYi4dSzwbG3_aJoSvwZo" found at
   _acme-challenge.omcr.io

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address.
```


Allow setting DNS TTL
Fixes #7969.

## Pull Request Checklist

- [x] If the change being made is to a [distributed component](https://certbot.eff.org/docs/contributing.html#code-components-and-layout), edit the `master` section of `certbot/CHANGELOG.md` to include a description of the change being made.
- [x] Include your name in `AUTHORS.md` if you like.

# Purpose
This pull request is to allow setting a custom TTL when using DigitalOcean TXT records for authentication, as well as to lower the default to a more reasonable 60 seconds.

# Rationale
The default used by the dependent `python-digitalocean` library is 1800 seconds (one-half hour) which can cause frustrations when testing out configurations with `--dry-run`.


Discussion:
Apparently it works if I specify `--dns-digitalocean-credentials /etc/digitalocean.ini` to the renew command. I cannot remember that I had to do that before. Is that changed? What credentials is it trying if I do not specify `--dns-digitalocean-credentials`?
I would guess that it's not `--dns-digitalocean-credentials` which made the difference, but rather that on one occasion, the default 10 second propagation delay used by the DigitalOcean DNS plugin was not sufficient (for some external reasons).

If this was a matter relating to the credentials file, then I would expect the DNS plugin to fail visibly with any number of errors (no credentials set, authentication failure, failure to find the DNS zone in your account). That doesn't appear to have been the case.

You could try experiment with `--dns-digitalocean-propagation-seconds XX --dry-run` to find a value where it reliably works (or doesn't), or just increase it to a more conservative e.g. 60 seconds.

10 seconds does seem to me to be a very optimistic default.

If this is related somehow to your credentials file after all, you can check what credentials file is being referenced by `/etc/letsencrypt/renewal/omcr.io.conf`.
it correctly says

```
dns_digitalocean_credentials = /etc/digitalocean.ini
```

The error message I got was not leading me to think this was a timing error. This is overall very strange but I think we might run into a "cannot reproduce" kind of situation. Let's see if anyone more picks up on this, that is using the dns-digitalocean plugin?
I still have a problem with this. I used the time-out of 60s and still I got an error. The system updates almost instantaneously (that's a tough word to spell; thanks spellcheck)  but certbot does not pickup on the change. Or rather it looks like the value set is not the expected value? What is going on here?
![image](https://user-images.githubusercontent.com/6200749/93019288-68e70b00-f5d6-11ea-950f-b987f33512da.png)
```
6c9928a10726# certbot certonly -d mq.horizon.omnimodular.net --dns-digitalocean-propagation-seconds 60 --dns-digitalocean-credentials /etc/digitalocean.ini 
Saving debug log to /var/log/letsencrypt/letsencrypt.log

How would you like to authenticate with the ACME CA?
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1: Obtain certs using a DNS TXT record (if you are using DigitalOcean for DNS).
(dns-digitalocean)
2: Spin up a temporary webserver (standalone)
3: Place files in webroot directory (webroot)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Select the appropriate number [1-3] then [enter] (press 'c' to cancel): 1
Plugins selected: Authenticator dns-digitalocean, Installer None
Obtaining a new certificate
Performing the following challenges:
dns-01 challenge for mq.horizon.omnimodular.net
Waiting 60 seconds for DNS changes to propagate
Waiting for verification...
Challenge failed for domain mq.horizon.omnimodular.net
dns-01 challenge for mq.horizon.omnimodular.net
Cleaning up challenges
Some challenges have failed.

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: mq.horizon.omnimodular.net
   Type:   unauthorized
   Detail: Incorrect TXT record
   "smljta9NShjMUpFTYSNIabV2Miillw4WAYrykjjowtU" found at
   _acme-challenge.mq.horizon.omnimodular.net

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address.
6c9928a10726# 
```

OK, I think I found causality for the error. If I try and obtain a TLS certificate for a CNAME it does not work. I can only get a TLS cert for a A type domain. Is this expected? I want to think I did not have any problems with CNAME before.

Ping @alexzorin 
Unless you have a CNAME on `_acme-challenge.example.com`, or something silly like a wildcard CNAME record, I don't think that would be the problem.

>The system updates almost instantaneously (that's a tough word to spell; thanks spellcheck) but certbot does not pickup on the change

There's two levels of updates here.

The first is that that the DigitalOcean user interface shows the new TXT record. This is expected to be instantaneous. 

However, it doesn't guarantee that the DigitalOcean nameservers have actually loaded the new records. That's where the delay can come into play. The only way to check this is to do a DNS query directly to the DO nameservers.

If Let's Encrypt reports a wrong value, then it seems likely to me that it's a stale value from the previous run.

If you set the propagation seconds to 120s, *and* you make sure to wait at least 60 seconds between attempts, does it still report the wrong value?
Same problem. owncloud is a A record in this case.
The TXT record value is not the one reported by Google toolkit.
https://toolbox.googleapps.com/apps/dig/#TXT/

```
7a072070c59b# certbot certonly -d owncloud.omnimodular.net --dns-digitalocean-propagation-seconds 120 --dns-digitalocean-credentials /etc/digitalocean.ini
Saving debug log to /var/log/letsencrypt/letsencrypt.log

How would you like to authenticate with the ACME CA?
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1: Obtain certs using a DNS TXT record (if you are using DigitalOcean for DNS).
(dns-digitalocean)
2: Spin up a temporary webserver (standalone)
3: Place files in webroot directory (webroot)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Select the appropriate number [1-3] then [enter] (press 'c' to cancel): 1
Plugins selected: Authenticator dns-digitalocean, Installer None
Obtaining a new certificate
Performing the following challenges:
dns-01 challenge for owncloud.omnimodular.net
Waiting 120 seconds for DNS changes to propagate
Waiting for verification...
Challenge failed for domain owncloud.omnimodular.net
dns-01 challenge for owncloud.omnimodular.net
Cleaning up challenges
Some challenges have failed.

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: owncloud.omnimodular.net
   Type:   unauthorized
   Detail: Incorrect TXT record
   "rOQbHcpQtnRJqKcEBb5C0GJzla14bQfIo0f4fzaWrZk" found at
   _acme-challenge.owncloud.omnimodular.net

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address.
7a072070c59b# 
```
Waiting 5-10 minutes and running the command again gave the same cached response "rOQbHcpQtnRJqKcEBb5C0GJzla14bQfIo0f4fzaWrZk".
Looking at the created TXT record at DO I see that the TTL is 1800 seconds so I guess I have to wait that long for the Let's encrypt service to throw away the cache? I do not see why it failed to begin with though since I used a 120 second propagation delay.

I tried running the same command (without waiting for 1800 seconds) in `certbot/dns-digitalocean:v1.7.0` and then it worked.
OK, I can reproduce this, but it isn't quite what either of us thought. Actually, this is inspired by [a recent thread on the community forums where I also struggled to understand what was happening with DigitalOcean](https://community.letsencrypt.org/t/certbot-renew-failing-to-renew-my-certificates-servfail-looking-up-a-for-my-site/134973).

So, to establish some facts:

- Currently, Certbot creates its TXT records with an 1800 TTL.
- Let's Encrypt's DNS recursors *do* observe TTLs, but they are clamped to 60 seconds. Meaning:
  -  if you post a record with 1800 TTL, Let's Encrypt will only cache it for 60 seconds.
  - if you post a record with 1 TTL, Let's Encrypt will only cache it for 1 second.

My  theory is that DigitalOcean now appears to now use some kind of anycast transparent DNS caching, powered by Cloudflare.

This really makes debugging a pain, as you have discovered:

> The TXT record value is not the one reported by Google toolkit.

I think I now understand what you are talking about. I wait 60 seconds after my first successful attempt (to account for Let's Encrypt's DNS cache to drop off), but I still get the "Incorrect TXT record". It seems impossible and goes completely against my understanding of Let's Encrypt's DNS ops.

In fact, it is not Let's Encrypt that is caching the TXT record. It is DigitalOcean's own anycast DNS cache which sits in front of their authoritative servers.

I am reliably triggering situations where I can perform the same query to e.g. `ns1.digitalocean.com` from my laptop, from unboundtest.com and from another server of mine, and I get 3 completely different answers, depending on what the state of the cache is at the anycast POP.

We have one important way to control this additional cache that DO is using: the TTL. From what I've observed, the transparent cache respects TTL. The problem is that we're feeding it 1800s, so any non-negatve responses are actually sitting in the edge caches for that full duration.

I've tried fixing this in the plugin locally (setting `ttl=30`, the published minimum), and it seems to fix the issue entirely. 

The only problem is that we would need to bump `python-digitalocean` to `1.12` (from the current `1.11` we require). Luckily, `ttl` is a kwarg, so it is possible to do this in a backwards-compatible way, if there are any distros who do not ship it.

@bmw do you have an opinion on whether it's worth unfreezing DNS to fix this? The impact is: users can only successfully use the plugin once every 30 minutes (TTL) per domain.
Thanks for investigating and writing this up @alexzorin.

I think updating the minimum version of `python-digitalocean` we require and the version we pin to a newer version is OK. Checking CentOS, Debian, and Ubuntu, the oldest version of this library I can find in use is 1.13.2.

I think that in the scope of all Certbot issues, the impact fixing this will have is pretty low, however, it shouldn't be that hard and it is something that should be done. The hardest part is updating the pinnings in a way that they still work everywhere we need them to. The way I'd personally do this is see if we can get away with just updating `python-digitalocean` by checking for dependency changes between versions. There is more discussion about the potential problems that can be hit here at https://github.com/certbot/certbot/issues/8290.
>  I do not see why it failed to begin with though since I used a 120 second propagation delay.

I remember now that I used a "dry-run" prior to the real attempt. That could have triggered the DNS cache issue you described. 
I was mistaken about the version required - we'd need `1.15`. Bummer.
That's unfortunate. We could potentially just bump the dependency anyway since we're taking more control of our packaging and the Debian/Ubuntu packages in released versions of those distros are not seeing any regular updates, but it's something we've avoided doing in the past.
I agree with the rationale. Consecutive dry-runs within the TTL period can produce confusing results, depending on how the ACME server observes TTLs.

If I can make a suggestion: we may as well lower it all the way to 30s, which is the documented minimum for the DO API.  

It's too bad it can't be lowered to 0s/1s, which would prevent the problem entirely. These are one-shot records, after all.
It's been 6 months, just because this is unplanned doesn't mean the PR should be left to sit.

It provides a much more useful default, especially for staging, preventing you from having to wait 30 minutes to have a real cert issued.

@bmw please merge
Any reason this hasn't been merged 1 year later?
Yeah I'm genuinely confused as to why this hasn't been merged. Is the change undesired? Or is it simply a matter of no one being available to review and approve it?
I'm already using a modified version of Certbot with this change and it works perfectly.
Hello,

The change is complicated by a dependency issue.

The `ttl` kwarg made an appearance in https://github.com/koalalorenzo/python-digitalocean/commit/c2071bc88a9e29e7487eecd0624a86ab0a5cc7da, released in `1.15`. Certbot only requires `>=1.11`. 

On a released stable distro like Debian, that only ships `1.13`, this PR has no effect. (And without version pins in `dev_constraints.txt` being updated as well, it would have no effect in Docker or the Certbot snap either).

Per [this comment](https://github.com/certbot/certbot/issues/8161#issuecomment-705724583), we could try to bump the required and pinned versions to `1.15` anyway.

Last time I had a crack at doing that, I noticed in https://github.com/koalalorenzo/python-digitalocean/compare/v1.11..v1.15.0 that an extra dependency on `jsonpickle` is introduced, and to avoid that I thought maybe to rewrite this provider using Lexicon, which Certbot already uses for other providers (though we'd need to bump the dependency there as well).

Since more users are hitting this, I will try to get an updated answer on how to progress this.
> On a released stable distro like Debian, that only ships `1.13`, this PR has no effect.

So? It doesn't hurt anything, it only helps people with a recent version. If you merge it you just have to make sure people know to upgrade their digitalocean package using `pip` if they need to use TTL, which is kind of how I setup everything in the first place.

You can rewrite the provider later :) Doesn't look like a blocker for this PR specifically.

Resolving patch:
diff --git a/certbot-dns-digitalocean/certbot_dns_digitalocean/_internal/dns_digitalocean.py b/certbot-dns-digitalocean/certbot_dns_digitalocean/_internal/dns_digitalocean.py
--- a/certbot-dns-digitalocean/certbot_dns_digitalocean/_internal/dns_digitalocean.py
+++ b/certbot-dns-digitalocean/certbot_dns_digitalocean/_internal/dns_digitalocean.py
@@ -21,6 +21,7 @@ class Authenticator(dns_common.DNSAuthenticator):
 
     description = 'Obtain certificates using a DNS TXT record (if you are ' + \
                   'using DigitalOcean for DNS).'
+    ttl = 30
 
     def __init__(self, *args, **kwargs):
         super(Authenticator, self).__init__(*args, **kwargs)
@@ -45,7 +46,8 @@ def _setup_credentials(self):
         )
 
     def _perform(self, domain, validation_name, validation):
-        self._get_digitalocean_client().add_txt_record(domain, validation_name, validation)
+        self._get_digitalocean_client().add_txt_record(domain, validation_name, validation,
+                                                       self.ttl)
 
     def _cleanup(self, domain, validation_name, validation):
         self._get_digitalocean_client().del_txt_record(domain, validation_name, validation)
@@ -62,13 +64,15 @@ class _DigitalOceanClient:
     def __init__(self, token):
         self.manager = digitalocean.Manager(token=token)
 
-    def add_txt_record(self, domain_name, record_name, record_content):
+    def add_txt_record(self, domain_name: str, record_name: str, record_content: str,
+                       record_ttl: int):
         """
         Add a TXT record using the supplied information.
 
         :param str domain_name: The domain to use to associate the record with.
         :param str record_name: The record name (typically beginning with '_acme-challenge.').
         :param str record_content: The record content (typically the challenge validation).
+        :param int record_ttl: The record TTL.
         :raises certbot.errors.PluginError: if an error occurs communicating with the DigitalOcean
                                             API
         """
@@ -89,7 +93,8 @@ def add_txt_record(self, domain_name, record_name, record_content):
             result = domain.create_new_domain_record(
                 type='TXT',
                 name=self._compute_record_name(domain, record_name),
-                data=record_content)
+                data=record_content,
+                ttl=record_ttl) # ttl kwarg is only effective starting python-digitalocean 1.15.0
 
             record_id = result['domain_record']['id']
 
@@ -99,7 +104,7 @@ def add_txt_record(self, domain_name, record_name, record_content):
             raise errors.PluginError('Error adding TXT record using the DigitalOcean API: {0}'
                                      .format(e))
 
-    def del_txt_record(self, domain_name, record_name, record_content):
+    def del_txt_record(self, domain_name: str, record_name: str, record_content: str):
         """
         Delete a TXT record using the supplied information.
 
diff --git a/certbot-dns-digitalocean/setup.py b/certbot-dns-digitalocean/setup.py
--- a/certbot-dns-digitalocean/setup.py
+++ b/certbot-dns-digitalocean/setup.py
@@ -9,7 +9,7 @@
 # Remember to update local-oldest-requirements.txt when changing the minimum
 # acme/certbot version.
 install_requires = [
-    'python-digitalocean>=1.11',
+    'python-digitalocean>=1.11', # 1.15.0 or newer is recommended for TTL support
     'setuptools>=39.0.1',
     'zope.interface',
 ]


Test patch:
diff --git a/certbot-dns-digitalocean/tests/dns_digitalocean_test.py b/certbot-dns-digitalocean/tests/dns_digitalocean_test.py
--- a/certbot-dns-digitalocean/tests/dns_digitalocean_test.py
+++ b/certbot-dns-digitalocean/tests/dns_digitalocean_test.py
@@ -40,7 +40,7 @@ def setUp(self):
     def test_perform(self):
         self.auth.perform([self.achall])
 
-        expected = [mock.call.add_txt_record(DOMAIN, '_acme-challenge.'+DOMAIN, mock.ANY)]
+        expected = [mock.call.add_txt_record(DOMAIN, '_acme-challenge.'+DOMAIN, mock.ANY, 30)]
         self.assertEqual(expected, self.mock_client.mock_calls)
 
     def test_cleanup(self):
@@ -58,6 +58,7 @@ class DigitalOceanClientTest(unittest.TestCase):
     record_prefix = "_acme-challenge"
     record_name = record_prefix + "." + DOMAIN
     record_content = "bar"
+    record_ttl = 60
 
     def setUp(self):
         from certbot_dns_digitalocean._internal.dns_digitalocean import _DigitalOceanClient
@@ -78,25 +79,27 @@ def test_add_txt_record(self):
 
         self.manager.get_all_domains.return_value = [wrong_domain_mock, domain_mock]
 
-        self.digitalocean_client.add_txt_record(DOMAIN, self.record_name, self.record_content)
+        self.digitalocean_client.add_txt_record(DOMAIN, self.record_name, self.record_content,
+                                                self.record_ttl)
 
         domain_mock.create_new_domain_record.assert_called_with(type='TXT',
                                                                 name=self.record_prefix,
-                                                                data=self.record_content)
+                                                                data=self.record_content,
+                                                                ttl=self.record_ttl)
 
     def test_add_txt_record_fail_to_find_domain(self):
         self.manager.get_all_domains.return_value = []
 
         self.assertRaises(errors.PluginError,
                           self.digitalocean_client.add_txt_record,
-                          DOMAIN, self.record_name, self.record_content)
+                          DOMAIN, self.record_name, self.record_content, self.record_ttl)
 
     def test_add_txt_record_error_finding_domain(self):
         self.manager.get_all_domains.side_effect = API_ERROR
 
         self.assertRaises(errors.PluginError,
                           self.digitalocean_client.add_txt_record,
-                          DOMAIN, self.record_name, self.record_content)
+                          DOMAIN, self.record_name, self.record_content, self.record_ttl)
 
     def test_add_txt_record_error_creating_record(self):
         domain_mock = mock.MagicMock()
@@ -107,7 +110,7 @@ def test_add_txt_record_error_creating_record(self):
 
         self.assertRaises(errors.PluginError,
                           self.digitalocean_client.add_txt_record,
-                          DOMAIN, self.record_name, self.record_content)
+                          DOMAIN, self.record_name, self.record_content, self.record_ttl)
 
     def test_del_txt_record(self):
         first_record_mock = mock.MagicMock()