Repository: certbot/certbot

Issue:
dns-cloudflare requires too many permissions
The `dns-cloudflare` module should provide support for manually specifying the `zone_id`. This quote from the documentation is well known:

> However, due to some shortcomings in Cloudflare’s implementation of Tokens, Tokens created for Certbot currently require Zone:Zone:Read and Zone:DNS:Edit permissions for **all** zones in your account. While this is not ideal, your Token will still have fewer permission than the Global key, so it’s still worth doing. Hopefully Cloudflare will improve this in the future.

You do not really need to give the token rights for **ALL** zones. You can successfully generate certificates with cloudflare if you use `manual-auth-hook`:
```
    certbot certonly \
      --manual \
      --non-interactive \
      --preferred-challenges dns \
      --manual-auth-hook "{{ work_folder }}/letsencrypt_auth.sh" \
      --manual-cleanup-hook "{{ work_folder }}/letsencrypt_cleanup.sh" \
      --rsa-key-size 4096 \
      --agree-tos \
      --manual-public-ip-logging-ok \
      --no-eff-email \
      --email {{ letsencrypt_email }} \
      --domains "*.{{ fqdn }}"
```
And having the following in your `letsencrypt_auth.sh`:
```
#!/bin/bash
EMAIL="xxx"
ZONE_ID="xxx"
TYPE="TXT"
NAME="_acme-challenge.$( echo "$CERTBOT_DOMAIN" | cut -d"." -f1-2 )"
CONTENT="$CERTBOT_VALIDATION"
TTL="1"
curl -X POST "https://api.cloudflare.com/client/v4/zones/$ZONE_ID/dns_records/" \
    -H "Authorization: xxxxxxxx" \
    -H "Content-Type: application/json" \
    --data '{"type":"'"$TYPE"'","name":"'"$NAME"'","content":"'"$CONTENT"'","ttl":'"$TTL"'}' 
```


When running with `certbot certonly --dns-cloudflare`, you will get an error as follows:
> Unable to determine zone_id for xxx using zone names: ['xxx', 'xxx', 'xxx', 'xxx']. Please confirm that the domain name has been entered correctly and is already associated with the supplied Cloudflare account. The error from Cloudflare was: 0 Actor 'com.cloudflare.api.token.xxxx' requires permission 'com.cloudflare.api.account.zone.list' to list zones


This happens because certbot forces you to give it access to all the zones under the account, instead of allowing you to manually specify the zone id.


Discussion:
[In a number of comments](https://github.com/certbot/certbot/pull/7583#issuecomment-596871471) it's been suggested that adding a `zone_id` option is not user-friendly enough, and that Cloudflare should fix their API.

Personally I think that's a valid position to take. I [wrote the part of the acme.sh wiki docs where `CF_Zone_ID` is mentioned](https://github.com/acmesh-official/acme.sh/wiki/dnsapi#1-cloudflare-option), and it definitely reduced the readability of the instructions.

cc @Tugzrida 
> it's been suggested that adding a `zone_id` option is not user-friendly enough

I do not see how this is a problem. People that don't want to read extended documentation will not know that they can also manually specify the `zone_id`. The rest will still use the software as it is now. The problem is that advanced users do not have this option.
Hello! Sorry everything's been a bit delayed, although I think that's understandable in the current global situation.

I have talked with @bmw and he sees (and I agree) that adding a zone ID override option is a last resort for after pushing Cloudflare to fix the issue, for multiple reasons:

- It adds complexity to the codebase and docs (although I have thought of an implementation that wouldn't be *too* bad)
- An overhaul of how certbot does DNS challenges is in the pipeline, so any changes now will just have to be re-implemented with that overhaul
- Improving Cloudflare's token permissions benefits **all** users of their API, not just certbot

This was a few weeks ago and, again due to the current environment, I haven't gotten to talking with Cloudflare yet. I have just now drafted an email and will send it after the long weekend so that it doesn't get buried.

In the interim, as @gun1x mentions, a hook script can be used. I wrote a script when I was last working on this a few weeks ago if anyone wishes to use that as a temporary solution: https://gist.github.com/Tugzrida/06b9c6f49683ebc30c9fcf2ed11a399a

If neither Brad nor I can get any assurance from Cloudflare, then adding the override is definitely on the table. It would allow for the best security with how Cloudflare's API is structured, and that's of course where we want to be, but implementing a patch without at least trying to fix the underlying issue is a bit silly.

I'll again apologise for any delay I've added, however the problem is widely known and Cloudflare support is only one email away, particularly for Enterprise or Business subscribers(which I am not) who'll get faster attention from the support team.

Edit: I also noticed the other day that Cloudflare made a few additions to the Tokens page (time and IP address restrictions). Hopefully that means the system is fresh in someone's brain so things can go smoother.
I've contacted Cloudflare, so hopefully there should be some more information within a few days.
Haven't heard anything other than an acknowledgement of the ticket. I'll chase it up next week.

As an interesting note, [this thread](https://community.cloudflare.com/t/how-to-get-zone-id-with-zone-specific-api-key/105300/4) on Cloudflare's forum mentions excluding zones. I've tested this and it does work - eg. have Zone:Zone:Read and Zone:DNS:Edit with Include:All Zones, then add Exclude:Specific Zone:other_zone for every other zone. It works, but that's ugly and could be very painful if you have a lot of zones, and when you add new zones, they won't be excluded. It also prevents DNS edits on the excluded zones, but the zone details can still be listed. There's not a lot of information in there, but it's still not ideal.
@Tugzrida thank you for investing time into this ticket. You are right, excluding each zone is painful. At this point, `manual-auth-hook` still seems to be the easiest solution.

You meantioned 1 week ago that having this option added would "add complexity to codebase and to docs". I don't think that adding another phrase to the docs would increase complexity that much. Regarding the complexity of the code, how about fetching the zone_id from env or from args and calling [_find_zone_id(domain)](https://github.com/certbot/certbot/blob/fc7e5e8e6060d9e0df2e704a20103d5c0f456925/certbot-dns-cloudflare/certbot_dns_cloudflare/_internal/dns_cloudflare.py#L107) only if it does not exist:

```
if not zone_id:
     zone_id = self._find_zone_id(domain)
```
That's basically along the lines of the implementation that I thought of, which would only be a few more lines, however my second and third points still stand. I will increase the frequency of my chase-ups if they continue to take their time, but I thought I'd leave it for a week at first.
Just when I was about to pass it over to @bmw, I look to have gotten to the front of the L2 support queue!

They wanted a bit more detail which I've now sent. If that's ok, they'll talk with the API team about their plans for fixing the permissions. I'll keep you all updated.
I haven't taken the time to fully grok and test https://community.cloudflare.com/t/bug-zone-detail-by-name-requires-zone-list-permission/128042/14 yet, but did Cloudflare just make a change that helps us here?

EDIT: I just got an email from the Cloudflare engineers we reached out to about this saying the problem is resolved so I think they did! :tada:

If someone is able to test and validate this, I think it would be useful.

Resolving patch:
diff --git a/certbot-dns-cloudflare/certbot_dns_cloudflare/__init__.py b/certbot-dns-cloudflare/certbot_dns_cloudflare/__init__.py
--- a/certbot-dns-cloudflare/certbot_dns_cloudflare/__init__.py
+++ b/certbot-dns-cloudflare/certbot_dns_cloudflare/__init__.py
@@ -21,8 +21,8 @@
 -----------
 
 Use of this plugin requires a configuration file containing Cloudflare API
-credentials, obtained from your Cloudflare
-`account page <https://dash.cloudflare.com/profile/api-tokens>`_.
+credentials, obtained from your
+`Cloudflare dashboard <https://dash.cloudflare.com/?to=/:account/profile/api-tokens>`_.
 
 Previously, Cloudflare's "Global API Key" was used for authentication, however
 this key can access the entire Cloudflare API for all domains in your account,
@@ -31,11 +31,8 @@
 Cloudflare's newer API Tokens can be restricted to specific domains and
 operations, and are therefore now the recommended authentication option.
 
-However, due to some shortcomings in Cloudflare's implementation of Tokens,
-Tokens created for Certbot currently require ``Zone:Zone:Read`` and ``Zone:DNS:Edit``
-permissions for **all** zones in your account. While this is not ideal, your Token
-will still have fewer permission than the Global key, so it's still worth doing.
-Hopefully Cloudflare will improve this in the future.
+The Token needed by Certbot requires ``Zone:DNS:Edit`` permissions for only the
+zones you need certificates for.
 
 Using Cloudflare Tokens also requires at least version 2.3.1 of the ``cloudflare``
 python module. If the version that automatically installed with this plugin is
diff --git a/certbot-dns-cloudflare/certbot_dns_cloudflare/_internal/dns_cloudflare.py b/certbot-dns-cloudflare/certbot_dns_cloudflare/_internal/dns_cloudflare.py
--- a/certbot-dns-cloudflare/certbot_dns_cloudflare/_internal/dns_cloudflare.py
+++ b/certbot-dns-cloudflare/certbot_dns_cloudflare/_internal/dns_cloudflare.py
@@ -14,7 +14,7 @@
 
 logger = logging.getLogger(__name__)
 
-ACCOUNT_URL = 'https://dash.cloudflare.com/profile/api-tokens'
+ACCOUNT_URL = 'https://dash.cloudflare.com/?to=/:account/profile/api-tokens'
 
 
 @zope.interface.implementer(interfaces.IAuthenticator)
@@ -118,7 +118,7 @@ def add_txt_record(self, domain, record_name, record_content, record_ttl):
             code = int(e)
             hint = None
 
-            if code == 9109:
+            if code == 1009:
                 hint = 'Does your API token have "Zone:DNS:Edit" permissions?'
 
             logger.error('Encountered CloudFlareAPIError adding TXT record: %d %s', e, e)
@@ -210,11 +210,22 @@ def _find_zone_id(self, domain):
                 logger.debug('Found zone_id of %s for %s using name %s', zone_id, domain, zone_name)
                 return zone_id
 
-        raise errors.PluginError('Unable to determine zone_id for {0} using zone names: {1}. '
-                                'Please confirm that the domain name has been entered correctly '
-                                'and is already associated with the supplied Cloudflare account.{2}'
-                                .format(domain, zone_name_guesses, ' The error from Cloudflare was:'
-                                ' {0} {1}'.format(code, msg) if code is not None else ''))
+        if msg is not None:
+            if 'com.cloudflare.api.account.zone.list' in msg:
+                raise errors.PluginError('Unable to determine zone_id for {0} using zone names: '
+                                         '{1}. Please confirm that the domain name has been '
+                                         'entered correctly and your Cloudflare Token has access '
+                                         'to the domain.'.format(domain, zone_name_guesses))
+            else:
+                raise errors.PluginError('Unable to determine zone_id for {0} using zone names: '
+                                         '{1}. The error from Cloudflare was: {2} {3}.'
+                                         .format(domain, zone_name_guesses, code, msg))
+        else:
+            raise errors.PluginError('Unable to determine zone_id for {0} using zone names: '
+                                     '{1}. Please confirm that the domain name has been '
+                                     'entered correctly and is already associated with the '
+                                     'supplied Cloudflare account.'
+                                     .format(domain, zone_name_guesses))
 
     def _find_txt_record_id(self, zone_id, record_name, record_content):
         """


Test patch:
diff --git a/certbot-dns-cloudflare/tests/dns_cloudflare_test.py b/certbot-dns-cloudflare/tests/dns_cloudflare_test.py
--- a/certbot-dns-cloudflare/tests/dns_cloudflare_test.py
+++ b/certbot-dns-cloudflare/tests/dns_cloudflare_test.py
@@ -133,7 +133,7 @@ def test_add_txt_record(self):
     def test_add_txt_record_error(self):
         self.cf.zones.get.return_value = [{'id': self.zone_id}]
 
-        self.cf.zones.dns_records.post.side_effect = CloudFlare.exceptions.CloudFlareAPIError(9109, '', '')
+        self.cf.zones.dns_records.post.side_effect = CloudFlare.exceptions.CloudFlareAPIError(1009, '', '')
 
         self.assertRaises(
             errors.PluginError,
@@ -175,6 +175,12 @@ def test_add_txt_record_bad_creds(self):
             self.cloudflare_client.add_txt_record,
             DOMAIN, self.record_name, self.record_content, self.record_ttl)
 
+        self.cf.zones.get.side_effect = CloudFlare.exceptions.CloudFlareAPIError(0, 'com.cloudflare.api.account.zone.list', '')
+        self.assertRaises(
+            errors.PluginError,
+            self.cloudflare_client.add_txt_record,
+            DOMAIN, self.record_name, self.record_content, self.record_ttl)
+
     def test_del_txt_record(self):
         self.cf.zones.get.return_value = [{'id': self.zone_id}]
         self.cf.zones.dns_records.get.return_value = [{'id': self.record_id}]