Repository: go-acme/lego (language: go)

Issue:
route53: lego deleted all TXT records for root domain
### Welcome

- [X] Yes, I'm using a binary release within 2 latest releases.
- [X] Yes, I've searched similar issues on GitHub and didn't find any.
- [X] Yes, I've included all information below (version, config, etc).

### What did you expect to see?

We are using the latest traefik, and we were testing some new services and deployed traefik with docker-compose. Our setup looks like this:

<details>

```yml
traefik:
  image: traefik
  ports:
   - 80:80
   - 8080:8080
   - 443:443
  command:
   - "--api.insecure=true"
   - "--providers.docker=true"
   - "--providers.docker.exposedbydefault=false"
   - "--entrypoints.web.address=:80"
   - "--entrypoints.web-secure.address=:443"
   - "--serverstransport.insecureskipverify=true"
   - "--certificatesresolvers.dnsresolver.acme.email=devops+letsencrypt@somedomain.com"
   - "--certificatesresolvers.dnsresolver.acme.storage=acme.json"
   - "--certificatesresolvers.dnsresolver.acme.dnschallenge.provider=route53"
   - "--certificatesresolvers.dnsresolver.acme.dnschallenge.delaybeforecheck=10"
   - "--certificatesresolvers.dnsresolver.acme.dnschallenge.resolvers=1.1.1.1:53,8.8.8.8:53"
```

</details>

And these are service labels:

<details>

```yml
labels:
   - "app=grafana"
   - "traefik.enable=true"
   - "traefik.docker.network=web"
   - "traefik.http.routers.grafana-somedomain.entrypoints=web"
   - "traefik.http.routers.grafana-somedomain.rule=Host(`logs.test.somedomain.com`)"
   - "traefik.http.routers.grafana-somedomain-https.rule=Host(`logs.test.somedomain.com`)"
   - "traefik.http.routers.grafana-somedomain-https.entrypoints=web-secure"
   - "traefik.http.routers.grafana-somedomain-https.tls=true"
   - "traefik.http.routers.grafana-somedomain-https.tls.certResolver=dnsresolver"
   - "traefik.http.routers.grafana-somedomain-https.tls.domains[0].main=logs.test.somedomain.com"
   - "traefik.http.services.grafana-somedomain-https.loadbalancer.server.port=3000"
```

</details>

We noticed that we are missing our root domain TXT records. Lego added the record that it needs, and after it verified the domain, it deleted all the others root domain TXT records. Just to mention, as you see from the labels, we didn't ask for the root domain certificate, nor any wildcard one, just for that one subdomain.
We caught that it was the lego library from the Cloudtrail logs:

<details>

```json
{
  "eventVersion": "1.08",
  "userIdentity": {
    "type": "AssumedRole",
    "principalId": "XXX",
    "arn": "arn:aws:sts::XXX:assumed-role/XXX/XXX",
    "accountId": "XXX",
    "accessKeyId": "XXX",
    "sessionContext": {
      "sessionIssuer": {
        "type": "Role",
        "principalId": "XXX",
        "arn": "arn:aws:iam::XXX:role/XXX",
        "accountId": "XXX",
        "userName": "XXX"
      },
      "webIdFederationData": {},
      "attributes": {
        "creationDate": "2022-11-15T10:50:05Z",
        "mfaAuthenticated": "false"
      },
      "ec2RoleDelivery": "1.0"
    }
  },
  "eventTime": "2022-11-15T10:52:31Z",
  "eventSource": "route53.amazonaws.com",
  "eventName": "ChangeResourceRecordSets",
  "awsRegion": "us-east-1",
  "sourceIPAddress": "XXX",
  "userAgent": "aws-sdk-go/1.44.47 (go1.19.2; linux; arm64)",
  "requestParameters": {
    "hostedZoneId": "XXX",
    "changeBatch": {
      "comment": "Managed by Lego",
      "changes": [
        {
          "action": "UPSERT",
          "resourceRecordSet": {
            "name": "somedomain.com.",
            "type": "TXT",
            "tTL": 10,
            "resourceRecords": [
              {
                "value": "\"v=spf1 include:_spf.google.com include:mail.zendesk.com include:spf.mandrillapp.com include:amazonses.com -all\""
              },
              {
                "value": "\"XXX""
              },
              {
                "value": "\"google-site-verification=XXX\""
              },
              {
                "value": "\"yandex-verification: XXX\""
              },
              {
                "value": "\"stripe-verification=XXX\""
              },
              {
                "value": "\"MS=XXX\""
              },
              {
                "value": "\"google-site-verification=XXX\""
              },
              {
                "value": "\"apple-domain-verification=XXX\""
              },
              {
                "value": "\"google-site-verification=XXX\""
              },
              {
                "value": "\"facebook-domain-verification=XXX\""
              },
              {
                "value": "\"XXX\""
              }
            ]
          }
        }
      ]
    }
  },
  "responseElements": {
    "changeInfo": {
      "id": "/change/XXX",
      "status": "PENDING",
      "submittedAt": "Nov 15, 2022 10:52:31 AM",
      "comment": "Managed by Lego"
    }
  },
  "additionalEventData": {
    "Note": "Do not use to reconstruct hosted zone"
  },
  "requestID": "e5448e23-fed7-4438-a592-XXX",
  "eventID": "9b30b976-9176-491d-b16a-XXX",
  "readOnly": false,
  "eventType": "AwsApiCall",
  "apiVersion": "2013-04-01",
  "managementEvent": true,
  "recipientAccountId": "XXX",
  "eventCategory": "Management",
  "tlsDetails": {
    "tlsVersion": "TLSv1.2",
    "cipherSuite": "ECDHE-RSA-AES128-GCM-SHA256",
    "clientProvidedHostHeader": "route53.amazonaws.com"
  }
},
```
</details>

And this is where it deleted it all:

<details>

```json
{
      "eventVersion": "1.08",
      "userIdentity": {
        "type": "AssumedRole",
        "principalId": "XXX",
        "arn": "arn:aws:sts::XXX:assumed-role/XXX/XXX",
        "accountId": "XXX",
        "accessKeyId": "XXX",
        "sessionContext": {
          "sessionIssuer": {
            "type": "Role",
            "principalId": "XXX",
            "arn": "arn:aws:iam::XXX:role/XXX",
            "accountId": "XXX",
            "userName": "XXX"
          },
          "webIdFederationData": {},
          "attributes": {
            "creationDate": "2022-11-15T10:50:05Z",
            "mfaAuthenticated": "false"
          },
          "ec2RoleDelivery": "1.0"
        }
      },
      "eventTime": "2022-11-15T10:53:40Z",
      "eventSource": "route53.amazonaws.com",
      "eventName": "ChangeResourceRecordSets",
      "awsRegion": "us-east-1",
      "sourceIPAddress": "XXX",
      "userAgent": "aws-sdk-go/1.44.47 (go1.19.2; linux; arm64)",
      "requestParameters": {
        "hostedZoneId": "XXX",
        "changeBatch": {
          "comment": "Managed by Lego",
          "changes": [
            {
              "action": "DELETE",
              "resourceRecordSet": {
                "name": "somedomain.com.",
                "type": "TXT",
                "tTL": 10,
                "resourceRecords": [
                  {
                    "value": "\"v=spf1 include:_spf.google.com include:mail.zendesk.com include:spf.mandrillapp.com include:amazonses.com -all\""
                  },
                  {
                    "value": "\"ahrefs-site-XXX\""
                  },
                  {
                    "value": "\"google-site-verification=XXX\""
                  },
                  {
                    "value": "\"yandex-verification: XXX\""
                  },
                  {
                    "value": "\"stripe-verification=XXX\""
                  },
                  {
                    "value": "\"MS=XXX\""
                  },
                  {
                    "value": "\"google-site-verification=XXX\""
                  },
                  {
                    "value": "\"apple-domain-verification=XXX\""
                  },
                  {
                    "value": "\"google-site-verification=XXX\""
                  },
                  {
                    "value": "\"facebook-domain-verification=XXX\""
                  },
                  {
                    "value": "\"XXX\""
                  }
                ]
              }
            }
          ]
        }
      },
      "responseElements": {
        "changeInfo": {
          "id": "/change/XXX",
          "status": "PENDING",
          "submittedAt": "Nov 15, 2022 10:53:40 AM",
          "comment": "Managed by Lego"
        }
      },
      "additionalEventData": {
        "Note": "Do not use to reconstruct hosted zone"
      },
      "requestID": "210676d7-7a39-48ed-8604-XXX",
      "eventID": "c97381f8-3b1a-484c-a881-XXX",
      "readOnly": false,
      "eventType": "AwsApiCall",
      "apiVersion": "2013-04-01",
      "managementEvent": true,
      "recipientAccountId": "XXX",
      "eventCategory": "Management",
      "tlsDetails": {
        "tlsVersion": "TLSv1.2",
        "cipherSuite": "ECDHE-RSA-AES128-GCM-SHA256",
        "clientProvidedHostHeader": "route53.amazonaws.com"
      }
    },
```
</details>

Luckily we had a backup of everything and we added those records back. This is really serious issue...

### What did you see instead?

That Lego deletes only what it adds to TXT record.

### How do you use lego?

Through Traefik

### Reproduction steps

Traefik configuration is in "What did you expect to see"

### Version of lego

```console
Traefik v2.9.4
```


### Logs

<details>

Cloudtrail logs are in What did you expect to see

</details>

### Go environment (if applicable)

<details>

```console
$ go version && go env
# paste output here
```

</details>


Discussion / hints:
We had the same issue with google cloud dns.
The TXT Record from the root domain was completely removed.

We downgraded lego to 4.7.0 and it works.
Are you using a CNAME wildcard?
Can you provide more information about your context?
Have you tried v4.8?
No, not any wildcard. Just a clear subdomain, as you can see in traefik labels above.
@dzydzany are you able to reproducible it multiple times?
Have you tried other versions of Traefik or lego?

For me it is the same only subdomains.
```sh
lego --accept-tos --email="admin@example.org" --domains=test1.example.org --domains=test2.example.org --dns="gcloud" --path="$CERTS_PATH" --pem run
```
I tried the latest container image version goacme/lego
We are actively using it across multiple stages and projects, and single subdomain and wildcard. And it worked until this version without problems.
What is the latest working version?
Traefik I am finding in one of our envs is 2.8.7
v2.8.7 of Traefik uses v4.7.0 of lego.

https://github.com/go-acme/lego/compare/v4.7.0...v4.9.0

:thinking: The only changes we have are around CNAME support.

Can you try to set the env var `LEGO_DISABLE_CNAME_SUPPORT` to `true`? (I don't see the link between CNAME and your problem but it is the only real change inside the DNS challenge since v4.7.0)
Uh, I would need some testing domain for this to test. Will let you know once I test it.
FYI the code of Route53 inside lego hasn't changed since 2018.
@shortmann same thing for you:
- What is the latest working version?
- Are you able to reproducible it multiple times?
- Do you have a wildcard CNAME DNS entry defined in your DNS provider?
- Can you try to set the env var `LEGO_DISABLE_CNAME_SUPPORT` to `true`?

FYI the code of Google Cloud inside lego hasn't changed since 2019.
I have a personal proverb: "If several people have a 404, it doesn't mean it is the same cause, it can but correlation doesn't mean causality."

So I'll wait for feedback from both of you.

For now, the only possibility that I can see is a CNAME side effect.

> @shortmann same thing for you:
> 
>     * What is the latest working version?
> 
>     * Are you able to reproducible it multiple times?
> 
>     * Do you have a wildcard CNAME DNS entry defined in your DNS provider?
> 
>     * Can you try to set the env var `LEGO_DISABLE_CNAME_SUPPORT` to `true`?
> 
> 
> FYI the code of Google Cloud inside lego hasn't changed since 2019.

It is related to CNAME support, as we do have wildcard CNAME.
is disabling CNAME support only way to go forward, or should we wait for some kind of fix?
Disabling CNAME is the only way.
I find a way to protect against unexpected deletions.
I will open a PR soon.

Fix patch:
diff --git a/providers/dns/route53/route53.go b/providers/dns/route53/route53.go
index b6e23757fa..e789aeb46b 100644
--- a/providers/dns/route53/route53.go
+++ b/providers/dns/route53/route53.go
@@ -176,26 +176,43 @@ func (d *DNSProvider) CleanUp(domain, token, keyAuth string) error {
 		return fmt.Errorf("failed to determine Route 53 hosted zone ID: %w", err)
 	}
 
-	records, err := d.getExistingRecordSets(ctx, hostedZoneID, info.EffectiveFQDN)
+	existingRecords, err := d.getExistingRecordSets(ctx, hostedZoneID, info.EffectiveFQDN)
 	if err != nil {
 		return fmt.Errorf("route53: %w", err)
 	}
 
-	if len(records) == 0 {
+	if len(existingRecords) == 0 {
 		return nil
 	}
 
+	var nonLegoRecords []awstypes.ResourceRecord
+	for _, record := range existingRecords {
+		if deref(record.Value) != `"`+info.Value+`"` {
+			nonLegoRecords = append(nonLegoRecords, record)
+		}
+	}
+
+	action := awstypes.ChangeActionUpsert
+
 	recordSet := &awstypes.ResourceRecordSet{
 		Name:            aws.String(info.EffectiveFQDN),
 		Type:            "TXT",
 		TTL:             aws.Int64(int64(d.config.TTL)),
-		ResourceRecords: records,
+		ResourceRecords: nonLegoRecords,
 	}
 
-	err = d.changeRecord(ctx, awstypes.ChangeActionDelete, hostedZoneID, recordSet)
+	// If the records are only records created by lego.
+	if len(nonLegoRecords) == 0 {
+		action = awstypes.ChangeActionDelete
+
+		recordSet.ResourceRecords = existingRecords
+	}
+
+	err = d.changeRecord(ctx, action, hostedZoneID, recordSet)
 	if err != nil {
 		return fmt.Errorf("route53: %w", err)
 	}
+
 	return nil
 }
 


Test patch: