Repository: certbot/certbot

Issue:
Certbot-dns-google fails with HTTP 412 - Precondition not met
## My operating system is (include version):
CentOS 7

## I installed Certbot with (certbot-auto, OS package manager, pip, etc):
yum install -y certbot
pip install certbot-dns-google

## I ran this command and it produced this output:

certbot certonly --dns-google --dns-google-credentials <path> --server https://acme-v02.api.letsencrypt.org/directory -d ssl1234.example.net -d *.example.net -d example.net  ...

## Certbot's behavior differed from what I expected because:

Encountered error adding TXT record: <HttpError 412 when requesting https://www.googleapis.com/dns/v1/projects/projectname/managedZones/id/changes?alt=json returned "Precondition not met for 'entity.change.deletions[0]'">

This problem only appeared if multiple domains are requested (around 15 SAN Names), single Domains work fine. I tested both wildcard domains and single domains - both fail sometimes. 

## Here is a Certbot log showing the issue (if available):
###### Logs are stored in `/var/log/letsencrypt` by default. Feel free to redact domains, e-mail and IP addresses as you see fit.

2018-05-24 16:05:21,163:WARNING:certbot_dns_google.dns_google:Encountered error deleting TXT record: <HttpError 412 when requesting https://www.googleapis.com/dns/v1/projects/projectname/managedZones/id/changes?alt=json returned "Precondition not met for 'entity.change.deletions[0]'">



Discussion:
Was just dealing with this and it turns out the issue for me was there was an existing txt record still in my domain's zone from when I had manually done the DNS challenge option the first time around. The API call used to delete that existing record is not providing the expected parameter for entity.change.deletions[0] so it's not able to delete the old entry.

I haven't dug into what exactly is expected in that API call so not sure what parameter isn't being provided correctly but you can work around this by deleting any old TXT records in the domain's zone that were used for the DNS challenge previously.

After that, certbot is able to add the TXT record and also removes it once it finishes so it shouldn't happen again.
I think this has something to do with subdomains that we use - we have one zone for example.org and also issue *.subdomain1.example.org and *.subdomain2.example.org - this works with the root domain and one subdomain but it seems to fail with another subdomain
It's been almost a year since this issue has seen any activity so I'm closing this issue. If you're still having this problem, please comment or open a new issue.
Hey, just ran into this myself here. I'm attempting to renew *.labs.shortbus.org which I did manually before, and it's failing with this same error.
Exactly the same issue, had a TXT record left around from the last challenge. When I removed "_acme-challenge.labs.shortbus.org." the google plugin worked again, but it seems it wasn't able to remove that record on its own, or it wasn't doing it correctly.
tl;dr for other lost souls ending up here: rigrassm's workaround works.
Just hit it again - rigrassm's workaround works.
OK. I'm reopening the issue.

EDIT: If anyone is able to figure out what we can do on our end to avoid the problem, I would love to see a PR!
I did a quick dive into the module and noticed that the `TXT` record deletion is only performed during the [`_cleanup` stage.](https://github.com/certbot/certbot/blob/f8e097a06176edfd69cd7306b6c83e3946b47f60/certbot-dns-google/certbot_dns_google/dns_google.py#L72)
```
    def _cleanup(self, domain, validation_name, validation):
        self._get_google_client().del_txt_record(domain, validation_name, validation, self.ttl)
```

Since this issue is being caused by the TXT record existing prior to utilizing this module to make the changes to the zone, the `_perform` stage runs into the collision with the existing record and doesn't have the logic built in to handle that scenario.

```
    def _perform(self, domain, validation_name, validation):
        self._get_google_client().add_txt_record(domain, validation_name, validation, self.ttl)
```

I'd imagine the easiest route to resolving this issue here would be to add a check for an existing record to the `_perform` function and delete it if it finds one before moving on to adding the new record. 

Edit: After looking further into the `add_txt_record` function I do see that it's attaching a `deletions` section to the request which is supposed to remove the existing record.

Since the `_cleanup` stage seems to be working correctly with regards to removing the record that gets added by the module. It may be worth exploring the differences between the `del_txt_record` and `add_txt_record` logic.

While having a single API call being performed for each stage is ideal, it may be worth exploring splitting up the addition and deletion of records into two separate calls(delete the record, then if that is successful, add the new record). This may also be helpful in troubleshooting what exactly is breaking the deletion of the existing record if Google's API returns a different error message specific to the deletion.

I'm not using Google's DNS anymore due to my trial account running out so I don't have anything to test things with ATM. If no one is able to get to this, then next time I get some free time I can start up a trial on a different Google account and try to work out what's happening here.
We've made a lot of changes to Certbot since this issue was opened. If you still have this issue with an up-to-date version of Certbot, can you please add a comment letting us know? This helps us to better see what issues are still affecting our users. If there is no activity in the next 30 days, this issue will be automatically closed.

This issue has been closed due to lack of activity, but if you think it should be reopened, please open a new issue with a link to this one and we'll take a look.


Resolving patch:
diff --git a/certbot-dns-google/certbot_dns_google/_internal/dns_google.py b/certbot-dns-google/certbot_dns_google/_internal/dns_google.py
--- a/certbot-dns-google/certbot_dns_google/_internal/dns_google.py
+++ b/certbot-dns-google/certbot_dns_google/_internal/dns_google.py
@@ -240,9 +240,10 @@ def get_existing_txt_rrset(self, zone_id, record_name):
 
         """
         rrs_request = self.dns.resourceRecordSets()
-        request = rrs_request.list(managedZone=zone_id, project=self.project_id)
         # Add dot as the API returns absolute domains
         record_name += "."
+        request = rrs_request.list(project=self.project_id, managedZone=zone_id, name=record_name,
+                                   type="TXT")
         try:
             response = request.execute()
         except googleapiclient_errors.Error:
@@ -250,10 +251,8 @@ def get_existing_txt_rrset(self, zone_id, record_name):
                         "requesting a wildcard certificate, this might not work.")
             logger.debug("Error was:", exc_info=True)
         else:
-            if response:
-                for rr in response["rrsets"]:
-                    if rr["name"] == record_name and rr["type"] == "TXT":
-                        return rr["rrdatas"]
+            if response and response["rrsets"]:
+                return response["rrsets"][0]["rrdatas"]
         return None
 
     def _find_managed_zone_id(self, domain):


Test patch:
diff --git a/certbot-dns-google/tests/dns_google_test.py b/certbot-dns-google/tests/dns_google_test.py
--- a/certbot-dns-google/tests/dns_google_test.py
+++ b/certbot-dns-google/tests/dns_google_test.py
@@ -70,7 +70,7 @@ class GoogleClientTest(unittest.TestCase):
     zone = "ZONE_ID"
     change = "an-id"
 
-    def _setUp_client_with_mock(self, zone_request_side_effect):
+    def _setUp_client_with_mock(self, zone_request_side_effect, rrs_list_side_effect=None):
         from certbot_dns_google._internal.dns_google import _GoogleClient
 
         pwd = os.path.dirname(__file__)
@@ -86,9 +86,16 @@ def _setUp_client_with_mock(self, zone_request_side_effect):
         mock_mz.list.return_value.execute.side_effect = zone_request_side_effect
 
         mock_rrs = mock.MagicMock()
-        rrsets = {"rrsets": [{"name": "_acme-challenge.example.org.", "type": "TXT",
+        def rrs_list(project=None, managedZone=None, name=None, type=None):
+            response = {"rrsets": []}
+            if name == "_acme-challenge.example.org.":
+                response = {"rrsets": [{"name": "_acme-challenge.example.org.", "type": "TXT",
                               "rrdatas": ["\"example-txt-contents\""]}]}
-        mock_rrs.list.return_value.execute.return_value = rrsets
+            mock_return = mock.MagicMock()
+            mock_return.execute.return_value = response
+            mock_return.execute.side_effect = rrs_list_side_effect
+            return mock_return
+        mock_rrs.list.side_effect = rrs_list
         mock_changes = mock.MagicMock()
 
         client.dns.managedZones = mock.MagicMock(return_value=mock_mz)
@@ -287,12 +294,19 @@ def test_del_txt_record_error_during_delete(self, unused_credential_mock):
     @mock.patch('oauth2client.service_account.ServiceAccountCredentials.from_json_keyfile_name')
     @mock.patch('certbot_dns_google._internal.dns_google.open',
                 mock.mock_open(read_data='{"project_id": "' + PROJECT_ID + '"}'), create=True)
-    def test_get_existing(self, unused_credential_mock):
+    def test_get_existing_found(self, unused_credential_mock):
         client, unused_changes = self._setUp_client_with_mock(
             [{'managedZones': [{'id': self.zone}]}])
         # Record name mocked in setUp
         found = client.get_existing_txt_rrset(self.zone, "_acme-challenge.example.org")
         self.assertEqual(found, ["\"example-txt-contents\""])
+
+    @mock.patch('oauth2client.service_account.ServiceAccountCredentials.from_json_keyfile_name')
+    @mock.patch('certbot_dns_google._internal.dns_google.open',
+                mock.mock_open(read_data='{"project_id": "' + PROJECT_ID + '"}'), create=True)
+    def test_get_existing_not_found(self, unused_credential_mock):
+        client, unused_changes = self._setUp_client_with_mock(
+            [{'managedZones': [{'id': self.zone}]}])
         not_found = client.get_existing_txt_rrset(self.zone, "nonexistent.tld")
         self.assertEqual(not_found, None)
 
@@ -301,10 +315,7 @@ def test_get_existing(self, unused_credential_mock):
                 mock.mock_open(read_data='{"project_id": "' + PROJECT_ID + '"}'), create=True)
     def test_get_existing_fallback(self, unused_credential_mock):
         client, unused_changes = self._setUp_client_with_mock(
-            [{'managedZones': [{'id': self.zone}]}])
-        mock_execute = client.dns.resourceRecordSets.return_value.list.return_value.execute
-        mock_execute.side_effect = API_ERROR
-
+            [{'managedZones': [{'id': self.zone}]}], API_ERROR)
         rrset = client.get_existing_txt_rrset(self.zone, "_acme-challenge.example.org")
         self.assertFalse(rrset)