Repository: go-acme/lego (language: go)

Issue:
ionos: DNS records not removed
### Welcome

- [X] Yes, I'm using a binary release within 2 latest releases.
- [X] Yes, I've searched similar issues on GitHub and didn't find any.
- [X] Yes, I've included all information below (version, config, etc).

### What did you expect to see?

Removal of the created _acme_challenge DNS records after successful validation.

### What did you see instead?

After the successful validation of the dns01-challange the created TXT _acme_challange records are not removed. Even though the logs state "[INFO] [traefik.example.com] acme: Cleaning DNS-01 challenge"
After waiting two hours the IONOS web ui still shows the records. 
The same procedure with [certbot/dns-ionos](https://github.com/helgeerbe/certbot-dns-ionos) removes the records immediately.

### How do you use lego?

Through Traefik

### Reproduction steps

1. Start docker container with the following docker compose file:

```yml
version: '3'

services:
  traefik:
    image: traefik:v2.10
    restart: unless-stopped
    volumes:
      - "letsencrypt:/letsencrypt"
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
    environment:
      - TZ=Europe/Berlin
      - TRAEFIK_API=true
      - TRAEFIK_API_DISABLEDASHBOARDAD=true
      - TRAEFIK_GLOBAL_SENDANONYMOUSUSAGE=false
      - TRAEFI_PROVIDERS_DOCKER=true
      - TRAEFIK_PROVIDERS_DOCKER_EXPOSEDBYDEFAULT=false
      - TRAEFIK_PROVIDERS_DOCKER_NETWORK=proxy
      - TRAEFIK_PROVIDERS_DOCKER_WATCH=true
      - TRAEFIK_CERTIFICATESRESOLVERS_LE=true
      - TRAEFIK_CERTIFICATESRESOLVERS_LE_ACME_CASERVER=https://acme-staging-v02.api.letsencrypt.org/directory
      - TRAEFIK_CERTIFICATESRESOLVERS_LE_ACME_CERTIFICATESDURATION=2160
      - TRAEFIK_CERTIFICATESRESOLVERS_LE_ACME_DNSCHALLENGE=true
      - TRAEFIK_CERTIFICATESRESOLVERS_LE_ACME_DNSCHALLENGE_DELAYBEFORECHECK=10
      - TRAEFIK_CERTIFICATESRESOLVERS_LE_ACME_STORAGE=/letsencrypt/acme.json
      - TRAEFIK_LOG=true
      - TRAEFIK_LOG_LEVEL=DEBUG
      - TRAEFIK_ACCESSLOG=true
      - TRAEFIK_ENTRYPOINTS_WEB=true
      - TRAEFIK_ENTRYPOINTS_WEBSECURE=true
      - TRAEFIK_ENTRYPOINTS_WEB_ADDRESS=:80
      - TRAEFIK_ENTRYPOINTS_WEBSECURE_ADDRESS=:443
      - TRAEFIK_ENTRYPOINTS_WEB_HTTP_REDIRECTIONS_ENTRYPOINT_TO=websecure
      - TRAEFIK_ENTRYPOINTS_WEBSECURE_HTTP_TLS=true
      - TRAEFIK_ENTRYPOINTS_WEBSECURE_HTTP_TLS_CERTRESOLVER=le
      - TRAEFIK_CERTIFICATESRESOLVERS_LE_ACME_DNSCHALLENGE_PROVIDER=ionos
      - TRAEFIK_CERTIFICATESRESOLVERS_LE_ACME_DNSCHALLENGE_RESOLVERS=ns1***.ui-dns.com
      - IONOS_API_KEY=<API_KEY>
      - TRAEFIK_CERTIFICATESRESOLVERS_LE_ACME_EMAIL=me@example.com
    ports:
      - 80:80
      - 443:443
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.api.rule=Host(`traefik.example.com`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))"
      - "traefik.http.routers.api.service=api@internal"
      - "traefik.http.routers.api.middlewares=auth"
      - "traefik.http.routers.api.tls=true"
      - "traefik.http.routers.api.tls.certresolver=le"

volumes:
  letsencrypt:
```
2. Wait for certificate creation
3. Check web ui

### Version of lego

```console
The lego command is not available in the traefik docker image. I am using the following image.

Output of traefik version:

Version: 2.10.7
Codename: saintmarcelin
Go version: go1.21.5
Built: 2023-12-06T15:54:59Z
OS/Arch: linux/arm64
```


### Logs

<details>

```console
traefik-traefik-1  | time="2024-01-12T17:42:34+01:00" level=info msg="Configuration loaded from environment variables."
traefik-traefik-1  | time="2024-01-12T17:42:34+01:00" level=info msg="Traefik version 2.10.7 built on 2023-12-06T15:54:59Z"
traefik-traefik-1  | time="2024-01-12T17:42:34+01:00" level=debug msg="Static configuration loaded {\"global\":{\"checkNewVersion\":true},\"serversTransport\":{\"maxIdleConnsPerHost\":200},\"entryPoints\":{\"web\":{\"address\":\":80\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":\"10s\"},\"respondingTimeouts\":{\"idleTimeout\":\"3m0s\"}},\"forwardedHeaders\":{},\"http\":{\"redirections\":{\"entryPoint\":{\"to\":\"websecure\",\"scheme\":\"https\",\"permanent\":true,\"priority\":2147483646}}},\"http2\":{\"maxConcurrentStreams\":250},\"udp\":{\"timeout\":\"3s\"}},\"websecure\":{\"address\":\":443\",\"transport\":{\"lifeCycle\":{\"graceTimeOut\":\"10s\"},\"respondingTimeouts\":{\"idleTimeout\":\"3m0s\"}},\"forwardedHeaders\":{},\"http\":{\"tls\":{\"certResolver\":\"le\"}},\"http2\":{\"maxConcurrentStreams\":250},\"udp\":{\"timeout\":\"3s\"}}},\"providers\":{\"providersThrottleDuration\":\"2s\",\"docker\":{\"watch\":true,\"endpoint\":\"unix:///var/run/docker.sock\",\"defaultRule\":\"Host(`{{ normalize .Name }}`)\",\"network\":\"proxy\",\"swarmModeRefreshSeconds\":\"15s\"}},\"api\":{\"dashboard\":true,\"disableDashboardAd\":true},\"log\":{\"level\":\"DEBUG\",\"format\":\"common\"},\"accessLog\":{\"format\":\"common\",\"filters\":{},\"fields\":{\"defaultMode\":\"keep\",\"headers\":{\"defaultMode\":\"drop\"}}},\"certificatesResolvers\":{\"le\":{\"acme\":{\"email\":\"tim@example.com\",\"caServer\":\"https://acme-staging-v02.api.letsencrypt.org/directory\",\"storage\":\"/letsencrypt/acme.json\",\"keyType\":\"RSA4096\",\"certificatesDuration\":2160,\"dnsChallenge\":{\"provider\":\"ionos\",\"delayBeforeCheck\":\"10s\",\"resolvers\":[\"ns1086.ui-dns.com\"]}}}}}"
traefik-traefik-1  | time="2024-01-12T17:42:34+01:00" level=info msg="\nStats collection is disabled.\nHelp us improve Traefik by turning this feature on :)\nMore details on: https://doc.traefik.io/traefik/contributing/data-collection/\n"
traefik-traefik-1  | time="2024-01-12T17:42:34+01:00" level=info msg="Starting provider aggregator aggregator.ProviderAggregator"
traefik-traefik-1  | time="2024-01-12T17:42:34+01:00" level=debug msg="Starting TCP Server" entryPointName=web
traefik-traefik-1  | time="2024-01-12T17:42:34+01:00" level=debug msg="Starting TCP Server" entryPointName=websecure
traefik-traefik-1  | time="2024-01-12T17:42:34+01:00" level=info msg="Starting provider *traefik.Provider"
traefik-traefik-1  | time="2024-01-12T17:42:34+01:00" level=debug msg="*traefik.Provider provider configuration: {}"
traefik-traefik-1  | time="2024-01-12T17:42:34+01:00" level=info msg="Starting provider *docker.Provider"
traefik-traefik-1  | time="2024-01-12T17:42:34+01:00" level=debug msg="*docker.Provider provider configuration: {\"watch\":true,\"endpoint\":\"unix:///var/run/docker.sock\",\"defaultRule\":\"Host(`{{ normalize .Name }}`)\",\"network\":\"proxy\",\"swarmModeRefreshSeconds\":\"15s\"}"
traefik-traefik-1  | time="2024-01-12T17:42:34+01:00" level=info msg="Starting provider *acme.Provider"
traefik-traefik-1  | time="2024-01-12T17:42:34+01:00" level=debug msg="*acme.Provider provider configuration: {\"email\":\"tim@example.com\",\"caServer\":\"https://acme-staging-v02.api.letsencrypt.org/directory\",\"storage\":\"/letsencrypt/acme.json\",\"keyType\":\"RSA4096\",\"certificatesDuration\":2160,\"dnsChallenge\":{\"provider\":\"ionos\",\"delayBeforeCheck\":\"10s\",\"resolvers\":[\"ns1086.ui-dns.com\"]},\"ResolverName\":\"le\",\"store\":{},\"TLSChallengeProvider\":{},\"HTTPChallengeProvider\":{}}"
traefik-traefik-1  | time="2024-01-12T17:42:34+01:00" level=info msg="Starting provider *acme.ChallengeTLSALPN"
traefik-traefik-1  | time="2024-01-12T17:42:34+01:00" level=debug msg="*acme.ChallengeTLSALPN provider configuration: {}"
traefik-traefik-1  | time="2024-01-12T17:42:34+01:00" level=debug msg="Attempt to renew certificates \"720h0m0s\" before expiry and check every \"24h0m0s\"" ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" providerName=le.acme
traefik-traefik-1  | time="2024-01-12T17:42:34+01:00" level=info msg="Testing certificate renew..." ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" providerName=le.acme
traefik-traefik-1  | time="2024-01-12T17:42:34+01:00" level=debug msg="Configuration received: {\"http\":{\"routers\":{\"web-to-websecure\":{\"entryPoints\":[\"web\"],\"middlewares\":[\"redirect-web-to-websecure\"],\"service\":\"noop@internal\",\"rule\":\"HostRegexp(`{host:.+}`)\",\"priority\":2147483646}},\"services\":{\"api\":{},\"dashboard\":{},\"noop\":{}},\"middlewares\":{\"redirect-web-to-websecure\":{\"redirectScheme\":{\"scheme\":\"https\",\"port\":\"443\",\"permanent\":true}}},\"models\":{\"websecure\":{\"tls\":{\"certResolver\":\"le\"}}},\"serversTransports\":{\"default\":{\"maxIdleConnsPerHost\":200}}},\"tcp\":{},\"udp\":{},\"tls\":{}}" providerName=internal
traefik-traefik-1  | time="2024-01-12T17:42:34+01:00" level=debug msg="Configuration received: {\"http\":{},\"tcp\":{},\"udp\":{},\"tls\":{}}" providerName=le.acme
traefik-traefik-1  | time="2024-01-12T17:42:34+01:00" level=debug msg="Provider connection established with docker 24.0.7 (API 1.43)" providerName=docker
traefik-traefik-1  | time="2024-01-12T17:42:34+01:00" level=debug msg="Filtering disabled container" providerName=docker container=pihole-exporter-pihole-exporter-e2ab7ff46cb96dd131a4bd8bba7052e0d8490fd46ac548fd8fb1a2470bccb46d
traefik-traefik-1  | time="2024-01-12T17:42:34+01:00" level=debug msg="Filtering disabled container" providerName=docker container=wireguard-wireguard-226646bff38fadef2bbe4aa7aeb5a5c08454f16053d0ab0cf1de7089e01cc5d7
traefik-traefik-1  | time="2024-01-12T17:42:34+01:00" level=debug msg="Filtering disabled container" container=prometheus-prometheus-66e034b0bcb3cde15bc86c8ff54c057f59b23862a34f27538996343f5f55148a providerName=docker
traefik-traefik-1  | time="2024-01-12T17:42:34+01:00" level=debug msg="Filtering disabled container" providerName=docker container=watchtower-watchtower-586fd7ee6d9ad3f01b1ca17fbbd01bb947ee04b1a4bba4fe9656759c2d2fbc4d
traefik-traefik-1  | time="2024-01-12T17:42:34+01:00" level=debug msg="Filtering disabled container" providerName=docker container=homepage-homepage-31e296ed376de742b603ea22286361e0129f12afa9c0a5439312f6b7cd58e018
traefik-traefik-1  | time="2024-01-12T17:42:34+01:00" level=debug msg="Filtering disabled container" providerName=docker container=node-exporter-node-exporter-daf8a990960e96be9760a6cf6d4113be9f5f0a1cc0a283aeb3cbeefde60484fb
traefik-traefik-1  | time="2024-01-12T17:42:34+01:00" level=debug msg="Configuration received: {\"http\":{\"routers\":{\"api\":{\"middlewares\":[\"auth\"],\"service\":\"api@internal\",\"rule\":\"Host(`traefik.example.com`) \\u0026\\u0026 (PathPrefix(`/api`) || PathPrefix(`/dashboard`))\",\"tls\":{\"certResolver\":\"le\"}},\"grafana\":{\"entryPoints\":[\"websecure\"],\"service\":\"grafana\",\"rule\":\"Host(`grafana.example.com`)\",\"tls\":{\"certResolver\":\"le\"}},\"pihole\":{\"entryPoints\":[\"websecure\"],\"service\":\"pihole\",\"rule\":\"Host(`dns.example.com`)\",\"tls\":{\"certResolver\":\"le\"}}},\"services\":{\"grafana\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://172.31.0.4:3000\"}],\"passHostHeader\":true}},\"pihole\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://172.31.0.2:80\"}],\"passHostHeader\":true}},\"traefik-traefik\":{\"loadBalancer\":{\"servers\":[{\"url\":\"http://172.31.0.3:80\"}],\"passHostHeader\":true}}},\"middlewares\":{\"auth\":{\"basicAuth\":{\"users\":[\"api:$2a$10$HIV8gT9hoCe9Kgb.PbuBPeQX.NYFFHmhX8T7J6obFBPtikxXmNpqK\"]}}}},\"tcp\":{},\"udp\":{}}" providerName=docker
traefik-traefik-1  | time="2024-01-12T17:42:36+01:00" level=debug msg="No default certificate, fallback to the internal generated certificate" tlsStoreName=default
traefik-traefik-1  | time="2024-01-12T17:42:36+01:00" level=debug msg="Added outgoing tracing middleware noop@internal" routerName=web-to-websecure@internal middlewareName=tracing middlewareType=TracingForwarder entryPointName=web
traefik-traefik-1  | time="2024-01-12T17:42:36+01:00" level=debug msg="Creating middleware" entryPointName=web middlewareName=redirect-web-to-websecure@internal middlewareType=RedirectScheme routerName=web-to-websecure@internal
traefik-traefik-1  | time="2024-01-12T17:42:36+01:00" level=debug msg="Setting up redirection to https 443" routerName=web-to-websecure@internal entryPointName=web middlewareName=redirect-web-to-websecure@internal middlewareType=RedirectScheme
traefik-traefik-1  | time="2024-01-12T17:42:36+01:00" level=debug msg="Creating middleware" entryPointName=web middlewareName=traefik-internal-recovery middlewareType=Recovery
traefik-traefik-1  | time="2024-01-12T17:42:36+01:00" level=debug msg="No entryPoint defined for this router, using the default one(s) instead: [web websecure]" routerName=api
traefik-traefik-1  | time="2024-01-12T17:42:37+01:00" level=debug msg="No default certificate, fallback to the internal generated certificate" tlsStoreName=default
traefik-traefik-1  | time="2024-01-12T17:42:37+01:00" level=debug msg="Added outgoing tracing middleware noop@internal" middlewareType=TracingForwarder middlewareName=tracing entryPointName=web routerName=web-to-websecure@internal
traefik-traefik-1  | time="2024-01-12T17:42:37+01:00" level=debug msg="Creating middleware" entryPointName=web routerName=web-to-websecure@internal middlewareName=redirect-web-to-websecure@internal middlewareType=RedirectScheme
traefik-traefik-1  | time="2024-01-12T17:42:37+01:00" level=debug msg="Setting up redirection to https 443" middlewareName=redirect-web-to-websecure@internal middlewareType=RedirectScheme entryPointName=web routerName=web-to-websecure@internal
traefik-traefik-1  | time="2024-01-12T17:42:37+01:00" level=debug msg="Creating middleware" entryPointName=web middlewareType=Recovery middlewareName=traefik-internal-recovery
traefik-traefik-1  | time="2024-01-12T17:42:37+01:00" level=debug msg="Added outgoing tracing middleware api@internal" entryPointName=websecure routerName=websecure-api@docker middlewareName=tracing middlewareType=TracingForwarder
traefik-traefik-1  | time="2024-01-12T17:42:37+01:00" level=debug msg="Creating middleware" entryPointName=websecure routerName=websecure-api@docker middlewareName=auth@docker middlewareType=BasicAuth
traefik-traefik-1  | time="2024-01-12T17:42:37+01:00" level=debug msg="Adding tracing to middleware" entryPointName=websecure routerName=websecure-api@docker middlewareName=auth@docker
traefik-traefik-1  | time="2024-01-12T17:42:37+01:00" level=debug msg="Creating middleware" serviceName=grafana middlewareName=pipelining middlewareType=Pipelining routerName=grafana@docker entryPointName=websecure
traefik-traefik-1  | time="2024-01-12T17:42:37+01:00" level=debug msg="Creating load-balancer" entryPointName=websecure serviceName=grafana routerName=grafana@docker
traefik-traefik-1  | time="2024-01-12T17:42:37+01:00" level=debug msg="Creating server 0 http://172.31.0.4:3000" entryPointName=websecure serviceName=grafana serverName=0 routerName=grafana@docker
traefik-traefik-1  | time="2024-01-12T17:42:37+01:00" level=debug msg="child http://172.31.0.4:3000 now UP"
traefik-traefik-1  | time="2024-01-12T17:42:37+01:00" level=debug msg="Propagating new UP status"
traefik-traefik-1  | time="2024-01-12T17:42:37+01:00" level=debug msg="Added outgoing tracing middleware grafana" routerName=grafana@docker entryPointName=websecure middlewareName=tracing middlewareType=TracingForwarder
traefik-traefik-1  | time="2024-01-12T17:42:37+01:00" level=debug msg="Creating middleware" routerName=pihole@docker serviceName=pihole middlewareName=pipelining middlewareType=Pipelining entryPointName=websecure
traefik-traefik-1  | time="2024-01-12T17:42:37+01:00" level=debug msg="Creating load-balancer" routerName=pihole@docker serviceName=pihole entryPointName=websecure
traefik-traefik-1  | time="2024-01-12T17:42:37+01:00" level=debug msg="Creating server 0 http://172.31.0.2:80" entryPointName=websecure routerName=pihole@docker serverName=0 serviceName=pihole
traefik-traefik-1  | time="2024-01-12T17:42:37+01:00" level=debug msg="child http://172.31.0.2:80 now UP"
traefik-traefik-1  | time="2024-01-12T17:42:37+01:00" level=debug msg="Propagating new UP status"
traefik-traefik-1  | time="2024-01-12T17:42:37+01:00" level=debug msg="Added outgoing tracing middleware pihole" routerName=pihole@docker middlewareType=TracingForwarder middlewareName=tracing entryPointName=websecure
traefik-traefik-1  | time="2024-01-12T17:42:37+01:00" level=debug msg="Creating middleware" middlewareType=Recovery entryPointName=websecure middlewareName=traefik-internal-recovery
traefik-traefik-1  | time="2024-01-12T17:42:37+01:00" level=debug msg="Added outgoing tracing middleware api@internal" routerName=api@docker middlewareName=tracing middlewareType=TracingForwarder entryPointName=web
traefik-traefik-1  | time="2024-01-12T17:42:37+01:00" level=debug msg="Creating middleware" entryPointName=web routerName=api@docker middlewareName=auth@docker middlewareType=BasicAuth
traefik-traefik-1  | time="2024-01-12T17:42:37+01:00" level=debug msg="Adding tracing to middleware" entryPointName=web routerName=api@docker middlewareName=auth@docker
traefik-traefik-1  | time="2024-01-12T17:42:37+01:00" level=debug msg="Creating middleware" middlewareName=traefik-internal-recovery middlewareType=Recovery entryPointName=web
traefik-traefik-1  | time="2024-01-12T17:42:37+01:00" level=debug msg="Adding route for traefik.example.com with TLS options default" entryPointName=web
traefik-traefik-1  | time="2024-01-12T17:42:37+01:00" level=debug msg="Adding route for traefik.example.com with TLS options default" entryPointName=websecure
traefik-traefik-1  | time="2024-01-12T17:42:37+01:00" level=debug msg="Adding route for grafana.example.com with TLS options default" entryPointName=websecure
traefik-traefik-1  | time="2024-01-12T17:42:37+01:00" level=debug msg="Adding route for dns.example.com with TLS options default" entryPointName=websecure
traefik-traefik-1  | time="2024-01-12T17:42:37+01:00" level=debug msg="Trying to challenge certificate for domain [grafana.example.com] found in HostSNI rule" routerName=grafana@docker rule="Host(`grafana.example.com`)" providerName=le.acme ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory"
traefik-traefik-1  | time="2024-01-12T17:42:37+01:00" level=debug msg="Trying to challenge certificate for domain [dns.example.com] found in HostSNI rule" providerName=le.acme ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" routerName=pihole@docker rule="Host(`dns.example.com`)"
traefik-traefik-1  | time="2024-01-12T17:42:37+01:00" level=debug msg="Trying to challenge certificate for domain [traefik.example.com] found in HostSNI rule" providerName=le.acme ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" routerName=api@docker rule="Host(`traefik.example.com`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))"
traefik-traefik-1  | time="2024-01-12T17:42:37+01:00" level=debug msg="Trying to challenge certificate for domain [traefik.example.com] found in HostSNI rule" providerName=le.acme routerName=websecure-api@docker rule="Host(`traefik.example.com`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))" ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory"
traefik-traefik-1  | time="2024-01-12T17:42:37+01:00" level=debug msg="Looking for provided certificate(s) to validate [\"traefik.example.com\"]..." ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" providerName=le.acme routerName=websecure-api@docker rule="Host(`traefik.example.com`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))"
traefik-traefik-1  | time="2024-01-12T17:42:37+01:00" level=debug msg="Domains [\"traefik.example.com\"] need ACME certificates generation for domains \"traefik.example.com\"." rule="Host(`traefik.example.com`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))" ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" providerName=le.acme routerName=websecure-api@docker
traefik-traefik-1  | time="2024-01-12T17:42:37+01:00" level=debug msg="Loading ACME certificates [traefik.example.com]..." providerName=le.acme routerName=websecure-api@docker rule="Host(`traefik.example.com`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))" ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory"
traefik-traefik-1  | time="2024-01-12T17:42:37+01:00" level=debug msg="Looking for provided certificate(s) to validate [\"grafana.example.com\"]..." routerName=grafana@docker rule="Host(`grafana.example.com`)" providerName=le.acme ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory"
traefik-traefik-1  | time="2024-01-12T17:42:37+01:00" level=debug msg="Looking for provided certificate(s) to validate [\"dns.example.com\"]..." providerName=le.acme ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" routerName=pihole@docker rule="Host(`dns.example.com`)"
traefik-traefik-1  | time="2024-01-12T17:42:37+01:00" level=debug msg="Domains [\"dns.example.com\"] need ACME certificates generation for domains \"dns.example.com\"." ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" routerName=pihole@docker rule="Host(`dns.example.com`)" providerName=le.acme
traefik-traefik-1  | time="2024-01-12T17:42:37+01:00" level=debug msg="Loading ACME certificates [dns.example.com]..." ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" routerName=pihole@docker rule="Host(`dns.example.com`)" providerName=le.acme
traefik-traefik-1  | time="2024-01-12T17:42:37+01:00" level=debug msg="Looking for provided certificate(s) to validate [\"traefik.example.com\"]..." providerName=le.acme ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" routerName=api@docker rule="Host(`traefik.example.com`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))"
traefik-traefik-1  | time="2024-01-12T17:42:37+01:00" level=debug msg="Domains [\"grafana.example.com\"] need ACME certificates generation for domains \"grafana.example.com\"." rule="Host(`grafana.example.com`)" providerName=le.acme ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" routerName=grafana@docker
traefik-traefik-1  | time="2024-01-12T17:42:37+01:00" level=debug msg="Loading ACME certificates [grafana.example.com]..." providerName=le.acme ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" routerName=grafana@docker rule="Host(`grafana.example.com`)"
traefik-traefik-1  | time="2024-01-12T17:42:37+01:00" level=debug msg="No ACME certificate generation required for domains [\"traefik.example.com\"]." providerName=le.acme ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" routerName=api@docker rule="Host(`traefik.example.com`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))"
traefik-traefik-1  | time="2024-01-12T17:42:45+01:00" level=debug msg="Building ACME client..." providerName=le.acme
traefik-traefik-1  | time="2024-01-12T17:42:45+01:00" level=debug msg="https://acme-staging-v02.api.letsencrypt.org/directory" providerName=le.acme
traefik-traefik-1  | time="2024-01-12T17:42:45+01:00" level=info msg=Register... providerName=le.acme
traefik-traefik-1  | time="2024-01-12T17:42:45+01:00" level=debug msg="legolog: [INFO] acme: Registering account for tim@example.com"
traefik-traefik-1  | time="2024-01-12T17:42:46+01:00" level=debug msg="Using DNS Challenge provider: ionos" providerName=le.acme
traefik-traefik-1  | time="2024-01-12T17:42:46+01:00" level=debug msg="legolog: [INFO] [traefik.example.com] acme: Obtaining bundled SAN certificate"
traefik-traefik-1  | time="2024-01-12T17:42:46+01:00" level=debug msg="legolog: [INFO] [dns.example.com] acme: Obtaining bundled SAN certificate"
traefik-traefik-1  | time="2024-01-12T17:42:46+01:00" level=debug msg="legolog: [INFO] [grafana.example.com] acme: Obtaining bundled SAN certificate"
traefik-traefik-1  | time="2024-01-12T17:42:46+01:00" level=debug msg="legolog: [INFO] [traefik.example.com] AuthURL: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/10575331854"
traefik-traefik-1  | time="2024-01-12T17:42:46+01:00" level=debug msg="legolog: [INFO] [traefik.example.com] acme: Could not find solver for: tls-alpn-01"
traefik-traefik-1  | time="2024-01-12T17:42:46+01:00" level=debug msg="legolog: [INFO] [traefik.example.com] acme: Could not find solver for: http-01"
traefik-traefik-1  | time="2024-01-12T17:42:46+01:00" level=debug msg="legolog: [INFO] [traefik.example.com] acme: use dns-01 solver"
traefik-traefik-1  | time="2024-01-12T17:42:46+01:00" level=debug msg="legolog: [INFO] [traefik.example.com] acme: Preparing to solve DNS-01"
traefik-traefik-1  | time="2024-01-12T17:42:46+01:00" level=debug msg="legolog: [INFO] [dns.example.com] AuthURL: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/10575331864"
traefik-traefik-1  | time="2024-01-12T17:42:46+01:00" level=debug msg="legolog: [INFO] [dns.example.com] acme: Could not find solver for: tls-alpn-01"
traefik-traefik-1  | time="2024-01-12T17:42:46+01:00" level=debug msg="legolog: [INFO] [dns.example.com] acme: Could not find solver for: http-01"
traefik-traefik-1  | time="2024-01-12T17:42:46+01:00" level=debug msg="legolog: [INFO] [dns.example.com] acme: use dns-01 solver"
traefik-traefik-1  | time="2024-01-12T17:42:46+01:00" level=debug msg="legolog: [INFO] [dns.example.com] acme: Preparing to solve DNS-01"
traefik-traefik-1  | time="2024-01-12T17:42:46+01:00" level=debug msg="legolog: [INFO] [grafana.example.com] AuthURL: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/10575331884"
traefik-traefik-1  | time="2024-01-12T17:42:46+01:00" level=debug msg="legolog: [INFO] [grafana.example.com] acme: Could not find solver for: tls-alpn-01"
traefik-traefik-1  | time="2024-01-12T17:42:46+01:00" level=debug msg="legolog: [INFO] [grafana.example.com] acme: Could not find solver for: http-01"
traefik-traefik-1  | time="2024-01-12T17:42:46+01:00" level=debug msg="legolog: [INFO] [grafana.example.com] acme: use dns-01 solver"
traefik-traefik-1  | time="2024-01-12T17:42:46+01:00" level=debug msg="legolog: [INFO] [grafana.example.com] acme: Preparing to solve DNS-01"
traefik-traefik-1  | time="2024-01-12T17:42:48+01:00" level=debug msg="legolog: [INFO] [grafana.example.com] acme: Trying to solve DNS-01"
traefik-traefik-1  | time="2024-01-12T17:42:48+01:00" level=debug msg="legolog: [INFO] [grafana.example.com] acme: Checking DNS record propagation using [ns1086.ui-dns.com:53]"
traefik-traefik-1  | time="2024-01-12T17:42:48+01:00" level=debug msg="legolog: [INFO] [traefik.example.com] acme: Trying to solve DNS-01"
traefik-traefik-1  | time="2024-01-12T17:42:48+01:00" level=debug msg="legolog: [INFO] [traefik.example.com] acme: Checking DNS record propagation using [ns1086.ui-dns.com:53]"
traefik-traefik-1  | time="2024-01-12T17:42:48+01:00" level=debug msg="legolog: [INFO] [dns.example.com] acme: Trying to solve DNS-01"
traefik-traefik-1  | time="2024-01-12T17:42:48+01:00" level=debug msg="legolog: [INFO] [dns.example.com] acme: Checking DNS record propagation using [ns1086.ui-dns.com:53]"
traefik-traefik-1  | time="2024-01-12T17:42:50+01:00" level=debug msg="legolog: [INFO] Wait for propagation [timeout: 1m0s, interval: 2s]"
traefik-traefik-1  | time="2024-01-12T17:42:50+01:00" level=debug msg="Delaying 10000000000 rather than validating DNS propagation now." providerName=le.acme
traefik-traefik-1  | time="2024-01-12T17:42:50+01:00" level=debug msg="legolog: [INFO] Wait for propagation [timeout: 1m0s, interval: 2s]"
traefik-traefik-1  | time="2024-01-12T17:42:50+01:00" level=debug msg="Delaying 10000000000 rather than validating DNS propagation now." providerName=le.acme
traefik-traefik-1  | time="2024-01-12T17:42:50+01:00" level=debug msg="legolog: [INFO] Wait for propagation [timeout: 1m0s, interval: 2s]"
traefik-traefik-1  | time="2024-01-12T17:42:50+01:00" level=debug msg="Delaying 10000000000 rather than validating DNS propagation now." providerName=le.acme
traefik-traefik-1  | time="2024-01-12T17:43:00+01:00" level=debug msg="legolog: [INFO] [grafana.example.com] acme: Waiting for DNS record propagation."
traefik-traefik-1  | time="2024-01-12T17:43:01+01:00" level=debug msg="legolog: [INFO] [traefik.example.com] acme: Waiting for DNS record propagation."
traefik-traefik-1  | time="2024-01-12T17:43:01+01:00" level=debug msg="legolog: [INFO] [dns.example.com] acme: Waiting for DNS record propagation."
traefik-traefik-1  | time="2024-01-12T17:43:02+01:00" level=debug msg="Delaying 10000000000 rather than validating DNS propagation now." providerName=le.acme
traefik-traefik-1  | time="2024-01-12T17:43:03+01:00" level=debug msg="Delaying 10000000000 rather than validating DNS propagation now." providerName=le.acme
traefik-traefik-1  | time="2024-01-12T17:43:03+01:00" level=debug msg="Delaying 10000000000 rather than validating DNS propagation now." providerName=le.acme
traefik-traefik-1  | time="2024-01-12T17:43:13+01:00" level=debug msg="legolog: [INFO] [grafana.example.com] acme: Waiting for DNS record propagation."
traefik-traefik-1  | time="2024-01-12T17:43:13+01:00" level=debug msg="legolog: [INFO] [traefik.example.com] acme: Waiting for DNS record propagation."
traefik-traefik-1  | time="2024-01-12T17:43:13+01:00" level=debug msg="legolog: [INFO] [dns.example.com] acme: Waiting for DNS record propagation."
traefik-traefik-1  | time="2024-01-12T17:43:15+01:00" level=debug msg="Delaying 10000000000 rather than validating DNS propagation now." providerName=le.acme
traefik-traefik-1  | time="2024-01-12T17:43:15+01:00" level=debug msg="Delaying 10000000000 rather than validating DNS propagation now." providerName=le.acme
traefik-traefik-1  | time="2024-01-12T17:43:15+01:00" level=debug msg="Delaying 10000000000 rather than validating DNS propagation now." providerName=le.acme
traefik-traefik-1  | time="2024-01-12T17:43:25+01:00" level=debug msg="legolog: [INFO] [grafana.example.com] acme: Waiting for DNS record propagation."
traefik-traefik-1  | time="2024-01-12T17:43:25+01:00" level=debug msg="legolog: [INFO] [traefik.example.com] acme: Waiting for DNS record propagation."
traefik-traefik-1  | time="2024-01-12T17:43:25+01:00" level=debug msg="legolog: [INFO] [dns.example.com] acme: Waiting for DNS record propagation."
traefik-traefik-1  | time="2024-01-12T17:43:27+01:00" level=debug msg="Delaying 10000000000 rather than validating DNS propagation now." providerName=le.acme
traefik-traefik-1  | time="2024-01-12T17:43:27+01:00" level=debug msg="Delaying 10000000000 rather than validating DNS propagation now." providerName=le.acme
traefik-traefik-1  | time="2024-01-12T17:43:27+01:00" level=debug msg="Delaying 10000000000 rather than validating DNS propagation now." providerName=le.acme
traefik-traefik-1  | 172.31.0.1 - - [12/Jan/2024:16:43:31 +0000] "GET /admin/api.php?summaryRaw&overTimeData&topItems&recentItems&getQueryTypes&getForwardDestinations&getQuerySources&jsonForceObject&auth=bd24d470b6a8fa746351a51cf0cab81ea342403acb46443d6ec48de2b3fa0960 HTTP/1.1" 301 17 "-" "-" 1 "web-to-websecure@internal" "-" 0ms
traefik-traefik-1  | time="2024-01-12T17:43:37+01:00" level=debug msg="legolog: [INFO] [grafana.example.com] acme: Waiting for DNS record propagation."
traefik-traefik-1  | time="2024-01-12T17:43:37+01:00" level=debug msg="legolog: [INFO] [traefik.example.com] acme: Waiting for DNS record propagation."
traefik-traefik-1  | time="2024-01-12T17:43:37+01:00" level=debug msg="legolog: [INFO] [dns.example.com] acme: Waiting for DNS record propagation."
traefik-traefik-1  | time="2024-01-12T17:43:39+01:00" level=debug msg="Delaying 10000000000 rather than validating DNS propagation now." providerName=le.acme
traefik-traefik-1  | time="2024-01-12T17:43:39+01:00" level=debug msg="Delaying 10000000000 rather than validating DNS propagation now." providerName=le.acme
traefik-traefik-1  | time="2024-01-12T17:43:39+01:00" level=debug msg="Delaying 10000000000 rather than validating DNS propagation now." providerName=le.acme
traefik-traefik-1  | time="2024-01-12T17:44:00+01:00" level=debug msg="legolog: [INFO] [dns.example.com] The server validated our request"
traefik-traefik-1  | time="2024-01-12T17:44:00+01:00" level=debug msg="legolog: [INFO] [dns.example.com] acme: Cleaning DNS-01 challenge"
traefik-traefik-1  | time="2024-01-12T17:44:01+01:00" level=debug msg="legolog: [INFO] [dns.example.com] acme: Validations succeeded; requesting certificates"
traefik-traefik-1  | time="2024-01-12T17:44:03+01:00" level=debug msg="legolog: [INFO] [traefik.example.com] The server validated our request"
traefik-traefik-1  | time="2024-01-12T17:44:03+01:00" level=debug msg="legolog: [INFO] [traefik.example.com] acme: Cleaning DNS-01 challenge"
traefik-traefik-1  | time="2024-01-12T17:44:04+01:00" level=debug msg="legolog: [INFO] [grafana.example.com] The server validated our request"
traefik-traefik-1  | time="2024-01-12T17:44:04+01:00" level=debug msg="legolog: [INFO] [grafana.example.com] acme: Cleaning DNS-01 challenge"
traefik-traefik-1  | time="2024-01-12T17:44:06+01:00" level=debug msg="legolog: [INFO] [traefik.example.com] acme: Validations succeeded; requesting certificates"
traefik-traefik-1  | time="2024-01-12T17:44:06+01:00" level=debug msg="legolog: [INFO] [grafana.example.com] acme: Validations succeeded; requesting certificates"
traefik-traefik-1  | time="2024-01-12T17:44:14+01:00" level=debug msg="legolog: [INFO] Wait for certificate [timeout: 30s, interval: 500ms]"
traefik-traefik-1  | time="2024-01-12T17:44:15+01:00" level=debug msg="legolog: [INFO] [dns.example.com] Server responded with a certificate."
traefik-traefik-1  | time="2024-01-12T17:44:15+01:00" level=debug msg="Certificates obtained for domains [dns.example.com]" rule="Host(`dns.example.com`)" providerName=le.acme ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" routerName=pihole@docker
traefik-traefik-1  | time="2024-01-12T17:44:15+01:00" level=debug msg="Configuration received: {\"http\":{},\"tcp\":{},\"udp\":{},\"tls\":{}}" providerName=le.acme
traefik-traefik-1  | time="2024-01-12T17:44:15+01:00" level=debug msg="No entryPoint defined for this router, using the default one(s) instead: [web websecure]" routerName=api
traefik-traefik-1  | time="2024-01-12T17:44:15+01:00" level=debug msg="Adding certificate for domain(s) dns.example.com"
traefik-traefik-1  | time="2024-01-12T17:44:16+01:00" level=debug msg="No default certificate, fallback to the internal generated certificate" tlsStoreName=default
traefik-traefik-1  | time="2024-01-12T17:44:16+01:00" level=debug msg="Added outgoing tracing middleware noop@internal" entryPointName=web routerName=web-to-websecure@internal middlewareName=tracing middlewareType=TracingForwarder
traefik-traefik-1  | time="2024-01-12T17:44:16+01:00" level=debug msg="Creating middleware" entryPointName=web routerName=web-to-websecure@internal middlewareType=RedirectScheme middlewareName=redirect-web-to-websecure@internal
traefik-traefik-1  | time="2024-01-12T17:44:16+01:00" level=debug msg="Setting up redirection to https 443" routerName=web-to-websecure@internal middlewareType=RedirectScheme middlewareName=redirect-web-to-websecure@internal entryPointName=web
traefik-traefik-1  | time="2024-01-12T17:44:16+01:00" level=debug msg="Creating middleware" entryPointName=web middlewareName=traefik-internal-recovery middlewareType=Recovery
traefik-traefik-1  | time="2024-01-12T17:44:16+01:00" level=debug msg="Creating middleware" serviceName=grafana middlewareName=pipelining middlewareType=Pipelining entryPointName=websecure routerName=grafana@docker
traefik-traefik-1  | time="2024-01-12T17:44:16+01:00" level=debug msg="Creating load-balancer" serviceName=grafana entryPointName=websecure routerName=grafana@docker
traefik-traefik-1  | time="2024-01-12T17:44:16+01:00" level=debug msg="Creating server 0 http://172.31.0.4:3000" serverName=0 entryPointName=websecure routerName=grafana@docker serviceName=grafana
traefik-traefik-1  | time="2024-01-12T17:44:16+01:00" level=debug msg="child http://172.31.0.4:3000 now UP"
traefik-traefik-1  | time="2024-01-12T17:44:16+01:00" level=debug msg="Propagating new UP status"
traefik-traefik-1  | time="2024-01-12T17:44:16+01:00" level=debug msg="Added outgoing tracing middleware grafana" middlewareName=tracing middlewareType=TracingForwarder entryPointName=websecure routerName=grafana@docker
traefik-traefik-1  | time="2024-01-12T17:44:16+01:00" level=debug msg="Creating middleware" middlewareName=pipelining middlewareType=Pipelining serviceName=pihole entryPointName=websecure routerName=pihole@docker
traefik-traefik-1  | time="2024-01-12T17:44:16+01:00" level=debug msg="Creating load-balancer" routerName=pihole@docker serviceName=pihole entryPointName=websecure
traefik-traefik-1  | time="2024-01-12T17:44:16+01:00" level=debug msg="Creating server 0 http://172.31.0.2:80" routerName=pihole@docker serviceName=pihole serverName=0 entryPointName=websecure
traefik-traefik-1  | time="2024-01-12T17:44:16+01:00" level=debug msg="child http://172.31.0.2:80 now UP"
traefik-traefik-1  | time="2024-01-12T17:44:16+01:00" level=debug msg="Propagating new UP status"
traefik-traefik-1  | time="2024-01-12T17:44:16+01:00" level=debug msg="Added outgoing tracing middleware pihole" entryPointName=websecure routerName=pihole@docker middlewareName=tracing middlewareType=TracingForwarder
traefik-traefik-1  | time="2024-01-12T17:44:16+01:00" level=debug msg="Added outgoing tracing middleware api@internal" routerName=websecure-api@docker middlewareName=tracing middlewareType=TracingForwarder entryPointName=websecure
traefik-traefik-1  | time="2024-01-12T17:44:16+01:00" level=debug msg="Creating middleware" routerName=websecure-api@docker middlewareName=auth@docker middlewareType=BasicAuth entryPointName=websecure
traefik-traefik-1  | time="2024-01-12T17:44:16+01:00" level=debug msg="Adding tracing to middleware" routerName=websecure-api@docker middlewareName=auth@docker entryPointName=websecure
traefik-traefik-1  | time="2024-01-12T17:44:16+01:00" level=debug msg="Creating middleware" entryPointName=websecure middlewareName=traefik-internal-recovery middlewareType=Recovery
traefik-traefik-1  | time="2024-01-12T17:44:16+01:00" level=debug msg="Added outgoing tracing middleware api@internal" routerName=api@docker middlewareName=tracing middlewareType=TracingForwarder entryPointName=web
traefik-traefik-1  | time="2024-01-12T17:44:16+01:00" level=debug msg="Creating middleware" middlewareType=BasicAuth entryPointName=web routerName=api@docker middlewareName=auth@docker
traefik-traefik-1  | time="2024-01-12T17:44:16+01:00" level=debug msg="Adding tracing to middleware" entryPointName=web routerName=api@docker middlewareName=auth@docker
traefik-traefik-1  | time="2024-01-12T17:44:16+01:00" level=debug msg="Creating middleware" middlewareType=Recovery entryPointName=web middlewareName=traefik-internal-recovery
traefik-traefik-1  | time="2024-01-12T17:44:16+01:00" level=debug msg="Adding route for traefik.example.com with TLS options default" entryPointName=web
traefik-traefik-1  | time="2024-01-12T17:44:16+01:00" level=debug msg="Adding route for traefik.example.com with TLS options default" entryPointName=websecure
traefik-traefik-1  | time="2024-01-12T17:44:16+01:00" level=debug msg="Adding route for grafana.example.com with TLS options default" entryPointName=websecure
traefik-traefik-1  | time="2024-01-12T17:44:16+01:00" level=debug msg="Adding route for dns.example.com with TLS options default" entryPointName=websecure
traefik-traefik-1  | time="2024-01-12T17:44:16+01:00" level=debug msg="Trying to challenge certificate for domain [traefik.example.com] found in HostSNI rule" ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" rule="Host(`traefik.example.com`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))" routerName=api@docker providerName=le.acme
traefik-traefik-1  | time="2024-01-12T17:44:16+01:00" level=debug msg="Looking for provided certificate(s) to validate [\"traefik.example.com\"]..." rule="Host(`traefik.example.com`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))" routerName=api@docker providerName=le.acme ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory"
traefik-traefik-1  | time="2024-01-12T17:44:16+01:00" level=debug msg="Trying to challenge certificate for domain [traefik.example.com] found in HostSNI rule" providerName=le.acme ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" routerName=websecure-api@docker rule="Host(`traefik.example.com`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))"
traefik-traefik-1  | time="2024-01-12T17:44:16+01:00" level=debug msg="No ACME certificate generation required for domains [\"traefik.example.com\"]." providerName=le.acme ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" rule="Host(`traefik.example.com`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))" routerName=api@docker
traefik-traefik-1  | time="2024-01-12T17:44:16+01:00" level=debug msg="Trying to challenge certificate for domain [grafana.example.com] found in HostSNI rule" ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" rule="Host(`grafana.example.com`)" routerName=grafana@docker providerName=le.acme
traefik-traefik-1  | time="2024-01-12T17:44:16+01:00" level=debug msg="Looking for provided certificate(s) to validate [\"traefik.example.com\"]..." routerName=websecure-api@docker rule="Host(`traefik.example.com`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))" providerName=le.acme ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory"
traefik-traefik-1  | time="2024-01-12T17:44:16+01:00" level=debug msg="No ACME certificate generation required for domains [\"traefik.example.com\"]." routerName=websecure-api@docker rule="Host(`traefik.example.com`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))" providerName=le.acme ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory"
traefik-traefik-1  | time="2024-01-12T17:44:16+01:00" level=debug msg="Looking for provided certificate(s) to validate [\"grafana.example.com\"]..." routerName=grafana@docker providerName=le.acme ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" rule="Host(`grafana.example.com`)"
traefik-traefik-1  | time="2024-01-12T17:44:16+01:00" level=debug msg="No ACME certificate generation required for domains [\"grafana.example.com\"]." providerName=le.acme ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" rule="Host(`grafana.example.com`)" routerName=grafana@docker
traefik-traefik-1  | time="2024-01-12T17:44:16+01:00" level=debug msg="Trying to challenge certificate for domain [dns.example.com] found in HostSNI rule" providerName=le.acme ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" routerName=pihole@docker rule="Host(`dns.example.com`)"
traefik-traefik-1  | time="2024-01-12T17:44:16+01:00" level=debug msg="Looking for provided certificate(s) to validate [\"dns.example.com\"]..." providerName=le.acme ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" routerName=pihole@docker rule="Host(`dns.example.com`)"
traefik-traefik-1  | time="2024-01-12T17:44:16+01:00" level=debug msg="No ACME certificate generation required for domains [\"dns.example.com\"]." rule="Host(`dns.example.com`)" providerName=le.acme ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" routerName=pihole@docker
traefik-traefik-1  | time="2024-01-12T17:44:22+01:00" level=debug msg="legolog: [INFO] Wait for certificate [timeout: 30s, interval: 500ms]"
traefik-traefik-1  | time="2024-01-12T17:44:23+01:00" level=debug msg="legolog: [INFO] [traefik.example.com] Server responded with a certificate."
traefik-traefik-1  | time="2024-01-12T17:44:23+01:00" level=debug msg="Certificates obtained for domains [traefik.example.com]" routerName=websecure-api@docker rule="Host(`traefik.example.com`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))" ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" providerName=le.acme
traefik-traefik-1  | time="2024-01-12T17:44:23+01:00" level=debug msg="Configuration received: {\"http\":{},\"tcp\":{},\"udp\":{},\"tls\":{}}" providerName=le.acme
traefik-traefik-1  | time="2024-01-12T17:44:23+01:00" level=debug msg="No entryPoint defined for this router, using the default one(s) instead: [web websecure]" routerName=api
traefik-traefik-1  | time="2024-01-12T17:44:23+01:00" level=debug msg="Adding certificate for domain(s) dns.example.com"
traefik-traefik-1  | time="2024-01-12T17:44:23+01:00" level=debug msg="Adding certificate for domain(s) traefik.example.com"
traefik-traefik-1  | time="2024-01-12T17:44:24+01:00" level=debug msg="No default certificate, fallback to the internal generated certificate" tlsStoreName=default
traefik-traefik-1  | time="2024-01-12T17:44:24+01:00" level=debug msg="Added outgoing tracing middleware noop@internal" middlewareType=TracingForwarder entryPointName=web routerName=web-to-websecure@internal middlewareName=tracing
traefik-traefik-1  | time="2024-01-12T17:44:24+01:00" level=debug msg="Creating middleware" middlewareType=RedirectScheme entryPointName=web routerName=web-to-websecure@internal middlewareName=redirect-web-to-websecure@internal
traefik-traefik-1  | time="2024-01-12T17:44:24+01:00" level=debug msg="Setting up redirection to https 443" routerName=web-to-websecure@internal middlewareName=redirect-web-to-websecure@internal middlewareType=RedirectScheme entryPointName=web
traefik-traefik-1  | time="2024-01-12T17:44:24+01:00" level=debug msg="Creating middleware" middlewareName=traefik-internal-recovery middlewareType=Recovery entryPointName=web
traefik-traefik-1  | time="2024-01-12T17:44:24+01:00" level=debug msg="Creating middleware" serviceName=pihole middlewareName=pipelining middlewareType=Pipelining entryPointName=websecure routerName=pihole@docker
traefik-traefik-1  | time="2024-01-12T17:44:24+01:00" level=debug msg="Creating load-balancer" entryPointName=websecure routerName=pihole@docker serviceName=pihole
traefik-traefik-1  | time="2024-01-12T17:44:24+01:00" level=debug msg="Creating server 0 http://172.31.0.2:80" serviceName=pihole serverName=0 entryPointName=websecure routerName=pihole@docker
traefik-traefik-1  | time="2024-01-12T17:44:24+01:00" level=debug msg="child http://172.31.0.2:80 now UP"
traefik-traefik-1  | time="2024-01-12T17:44:24+01:00" level=debug msg="Propagating new UP status"
traefik-traefik-1  | time="2024-01-12T17:44:24+01:00" level=debug msg="Added outgoing tracing middleware pihole" routerName=pihole@docker middlewareName=tracing middlewareType=TracingForwarder entryPointName=websecure
traefik-traefik-1  | time="2024-01-12T17:44:24+01:00" level=debug msg="Added outgoing tracing middleware api@internal" middlewareName=tracing middlewareType=TracingForwarder entryPointName=websecure routerName=websecure-api@docker
traefik-traefik-1  | time="2024-01-12T17:44:24+01:00" level=debug msg="Creating middleware" entryPointName=websecure routerName=websecure-api@docker middlewareName=auth@docker middlewareType=BasicAuth
traefik-traefik-1  | time="2024-01-12T17:44:24+01:00" level=debug msg="Adding tracing to middleware" middlewareName=auth@docker entryPointName=websecure routerName=websecure-api@docker
traefik-traefik-1  | time="2024-01-12T17:44:24+01:00" level=debug msg="Creating middleware" routerName=grafana@docker middlewareType=Pipelining middlewareName=pipelining serviceName=grafana entryPointName=websecure
traefik-traefik-1  | time="2024-01-12T17:44:24+01:00" level=debug msg="Creating load-balancer" entryPointName=websecure routerName=grafana@docker serviceName=grafana
traefik-traefik-1  | time="2024-01-12T17:44:24+01:00" level=debug msg="Creating server 0 http://172.31.0.4:3000" serviceName=grafana serverName=0 entryPointName=websecure routerName=grafana@docker
traefik-traefik-1  | time="2024-01-12T17:44:24+01:00" level=debug msg="child http://172.31.0.4:3000 now UP"
traefik-traefik-1  | time="2024-01-12T17:44:24+01:00" level=debug msg="Propagating new UP status"
traefik-traefik-1  | time="2024-01-12T17:44:24+01:00" level=debug msg="Added outgoing tracing middleware grafana" middlewareName=tracing middlewareType=TracingForwarder entryPointName=websecure routerName=grafana@docker
traefik-traefik-1  | time="2024-01-12T17:44:24+01:00" level=debug msg="Creating middleware" entryPointName=websecure middlewareName=traefik-internal-recovery middlewareType=Recovery
traefik-traefik-1  | time="2024-01-12T17:44:24+01:00" level=debug msg="Added outgoing tracing middleware api@internal" middlewareName=tracing middlewareType=TracingForwarder entryPointName=web routerName=api@docker
traefik-traefik-1  | time="2024-01-12T17:44:24+01:00" level=debug msg="Creating middleware" routerName=api@docker middlewareName=auth@docker middlewareType=BasicAuth entryPointName=web
traefik-traefik-1  | time="2024-01-12T17:44:24+01:00" level=debug msg="Adding tracing to middleware" entryPointName=web routerName=api@docker middlewareName=auth@docker
traefik-traefik-1  | time="2024-01-12T17:44:24+01:00" level=debug msg="Creating middleware" entryPointName=web middlewareType=Recovery middlewareName=traefik-internal-recovery
traefik-traefik-1  | time="2024-01-12T17:44:24+01:00" level=debug msg="Adding route for traefik.example.com with TLS options default" entryPointName=web
traefik-traefik-1  | time="2024-01-12T17:44:24+01:00" level=debug msg="Adding route for dns.example.com with TLS options default" entryPointName=websecure
traefik-traefik-1  | time="2024-01-12T17:44:24+01:00" level=debug msg="Adding route for traefik.example.com with TLS options default" entryPointName=websecure
traefik-traefik-1  | time="2024-01-12T17:44:24+01:00" level=debug msg="Adding route for grafana.example.com with TLS options default" entryPointName=websecure
traefik-traefik-1  | time="2024-01-12T17:44:24+01:00" level=debug msg="Trying to challenge certificate for domain [traefik.example.com] found in HostSNI rule" ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" providerName=le.acme rule="Host(`traefik.example.com`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))" routerName=api@docker
traefik-traefik-1  | time="2024-01-12T17:44:24+01:00" level=debug msg="Trying to challenge certificate for domain [traefik.example.com] found in HostSNI rule" providerName=le.acme ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" routerName=websecure-api@docker rule="Host(`traefik.example.com`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))"
traefik-traefik-1  | time="2024-01-12T17:44:24+01:00" level=debug msg="Looking for provided certificate(s) to validate [\"traefik.example.com\"]..." providerName=le.acme rule="Host(`traefik.example.com`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))" routerName=api@docker ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory"
traefik-traefik-1  | time="2024-01-12T17:44:24+01:00" level=debug msg="Trying to challenge certificate for domain [grafana.example.com] found in HostSNI rule" rule="Host(`grafana.example.com`)" ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" providerName=le.acme routerName=grafana@docker
traefik-traefik-1  | time="2024-01-12T17:44:24+01:00" level=debug msg="No ACME certificate generation required for domains [\"traefik.example.com\"]." ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" providerName=le.acme rule="Host(`traefik.example.com`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))" routerName=api@docker
traefik-traefik-1  | time="2024-01-12T17:44:24+01:00" level=debug msg="Looking for provided certificate(s) to validate [\"traefik.example.com\"]..." routerName=websecure-api@docker rule="Host(`traefik.example.com`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))" providerName=le.acme ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory"
traefik-traefik-1  | time="2024-01-12T17:44:24+01:00" level=debug msg="Looking for provided certificate(s) to validate [\"grafana.example.com\"]..." rule="Host(`grafana.example.com`)" ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" providerName=le.acme routerName=grafana@docker
traefik-traefik-1  | time="2024-01-12T17:44:24+01:00" level=debug msg="Trying to challenge certificate for domain [dns.example.com] found in HostSNI rule" ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" providerName=le.acme routerName=pihole@docker rule="Host(`dns.example.com`)"
traefik-traefik-1  | time="2024-01-12T17:44:24+01:00" level=debug msg="No ACME certificate generation required for domains [\"traefik.example.com\"]." rule="Host(`traefik.example.com`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))" providerName=le.acme ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" routerName=websecure-api@docker
traefik-traefik-1  | time="2024-01-12T17:44:24+01:00" level=debug msg="No ACME certificate generation required for domains [\"grafana.example.com\"]." ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" providerName=le.acme routerName=grafana@docker rule="Host(`grafana.example.com`)"
traefik-traefik-1  | time="2024-01-12T17:44:24+01:00" level=debug msg="Looking for provided certificate(s) to validate [\"dns.example.com\"]..." providerName=le.acme routerName=pihole@docker rule="Host(`dns.example.com`)" ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory"
traefik-traefik-1  | time="2024-01-12T17:44:24+01:00" level=debug msg="No ACME certificate generation required for domains [\"dns.example.com\"]." providerName=le.acme routerName=pihole@docker rule="Host(`dns.example.com`)" ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory"
traefik-traefik-1  | 172.31.0.1 - - [12/Jan/2024:16:44:31 +0000] "GET /admin/api.php?summaryRaw&overTimeData&topItems&recentItems&getQueryTypes&getForwardDestinations&getQuerySources&jsonForceObject&auth=bd24d470b6a8fa746351a51cf0cab81ea342403acb46443d6ec48de2b3fa0960 HTTP/1.1" 301 17 "-" "-" 2 "web-to-websecure@internal" "-" 0ms
traefik-traefik-1  | time="2024-01-12T17:44:32+01:00" level=debug msg="legolog: [INFO] Wait for certificate [timeout: 30s, interval: 500ms]"
traefik-traefik-1  | time="2024-01-12T17:44:33+01:00" level=debug msg="legolog: [INFO] [grafana.example.com] Server responded with a certificate."
traefik-traefik-1  | time="2024-01-12T17:44:33+01:00" level=debug msg="Certificates obtained for domains [grafana.example.com]" rule="Host(`grafana.example.com`)" providerName=le.acme ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" routerName=grafana@docker
traefik-traefik-1  | time="2024-01-12T17:44:33+01:00" level=debug msg="Configuration received: {\"http\":{},\"tcp\":{},\"udp\":{},\"tls\":{}}" providerName=le.acme
traefik-traefik-1  | time="2024-01-12T17:44:33+01:00" level=debug msg="No entryPoint defined for this router, using the default one(s) instead: [web websecure]" routerName=api
traefik-traefik-1  | time="2024-01-12T17:44:33+01:00" level=debug msg="Adding certificate for domain(s) dns.example.com"
traefik-traefik-1  | time="2024-01-12T17:44:33+01:00" level=debug msg="Adding certificate for domain(s) traefik.example.com"
traefik-traefik-1  | time="2024-01-12T17:44:33+01:00" level=debug msg="Adding certificate for domain(s) grafana.example.com"
traefik-traefik-1  | time="2024-01-12T17:44:34+01:00" level=debug msg="No default certificate, fallback to the internal generated certificate" tlsStoreName=default
traefik-traefik-1  | time="2024-01-12T17:44:34+01:00" level=debug msg="Added outgoing tracing middleware noop@internal" middlewareType=TracingForwarder entryPointName=web routerName=web-to-websecure@internal middlewareName=tracing
traefik-traefik-1  | time="2024-01-12T17:44:34+01:00" level=debug msg="Creating middleware" routerName=web-to-websecure@internal entryPointName=web middlewareName=redirect-web-to-websecure@internal middlewareType=RedirectScheme
traefik-traefik-1  | time="2024-01-12T17:44:34+01:00" level=debug msg="Setting up redirection to https 443" routerName=web-to-websecure@internal entryPointName=web middlewareName=redirect-web-to-websecure@internal middlewareType=RedirectScheme
traefik-traefik-1  | time="2024-01-12T17:44:34+01:00" level=debug msg="Creating middleware" entryPointName=web middlewareName=traefik-internal-recovery middlewareType=Recovery
traefik-traefik-1  | time="2024-01-12T17:44:34+01:00" level=debug msg="Creating middleware" routerName=pihole@docker serviceName=pihole middlewareName=pipelining middlewareType=Pipelining entryPointName=websecure
traefik-traefik-1  | time="2024-01-12T17:44:34+01:00" level=debug msg="Creating load-balancer" serviceName=pihole entryPointName=websecure routerName=pihole@docker
traefik-traefik-1  | time="2024-01-12T17:44:34+01:00" level=debug msg="Creating server 0 http://172.31.0.2:80" serverName=0 entryPointName=websecure routerName=pihole@docker serviceName=pihole
traefik-traefik-1  | time="2024-01-12T17:44:34+01:00" level=debug msg="child http://172.31.0.2:80 now UP"
traefik-traefik-1  | time="2024-01-12T17:44:34+01:00" level=debug msg="Propagating new UP status"
traefik-traefik-1  | time="2024-01-12T17:44:34+01:00" level=debug msg="Added outgoing tracing middleware pihole" routerName=pihole@docker middlewareName=tracing middlewareType=TracingForwarder entryPointName=websecure
traefik-traefik-1  | time="2024-01-12T17:44:34+01:00" level=debug msg="Added outgoing tracing middleware api@internal" middlewareName=tracing middlewareType=TracingForwarder entryPointName=websecure routerName=websecure-api@docker
traefik-traefik-1  | time="2024-01-12T17:44:34+01:00" level=debug msg="Creating middleware" entryPointName=websecure routerName=websecure-api@docker middlewareName=auth@docker middlewareType=BasicAuth
traefik-traefik-1  | time="2024-01-12T17:44:34+01:00" level=debug msg="Adding tracing to middleware" entryPointName=websecure routerName=websecure-api@docker middlewareName=auth@docker
traefik-traefik-1  | time="2024-01-12T17:44:34+01:00" level=debug msg="Creating middleware" middlewareName=pipelining middlewareType=Pipelining entryPointName=websecure routerName=grafana@docker serviceName=grafana
traefik-traefik-1  | time="2024-01-12T17:44:34+01:00" level=debug msg="Creating load-balancer" routerName=grafana@docker serviceName=grafana entryPointName=websecure
traefik-traefik-1  | time="2024-01-12T17:44:34+01:00" level=debug msg="Creating server 0 http://172.31.0.4:3000" routerName=grafana@docker serviceName=grafana serverName=0 entryPointName=websecure
traefik-traefik-1  | time="2024-01-12T17:44:34+01:00" level=debug msg="child http://172.31.0.4:3000 now UP"
traefik-traefik-1  | time="2024-01-12T17:44:34+01:00" level=debug msg="Propagating new UP status"
traefik-traefik-1  | time="2024-01-12T17:44:34+01:00" level=debug msg="Added outgoing tracing middleware grafana" middlewareType=TracingForwarder entryPointName=websecure routerName=grafana@docker middlewareName=tracing
traefik-traefik-1  | time="2024-01-12T17:44:34+01:00" level=debug msg="Creating middleware" middlewareName=traefik-internal-recovery middlewareType=Recovery entryPointName=websecure
traefik-traefik-1  | time="2024-01-12T17:44:34+01:00" level=debug msg="Added outgoing tracing middleware api@internal" middlewareType=TracingForwarder entryPointName=web routerName=api@docker middlewareName=tracing
traefik-traefik-1  | time="2024-01-12T17:44:34+01:00" level=debug msg="Creating middleware" routerName=api@docker middlewareName=auth@docker middlewareType=BasicAuth entryPointName=web
traefik-traefik-1  | time="2024-01-12T17:44:34+01:00" level=debug msg="Adding tracing to middleware" entryPointName=web routerName=api@docker middlewareName=auth@docker
traefik-traefik-1  | time="2024-01-12T17:44:34+01:00" level=debug msg="Creating middleware" entryPointName=web middlewareName=traefik-internal-recovery middlewareType=Recovery
traefik-traefik-1  | time="2024-01-12T17:44:34+01:00" level=debug msg="Adding route for traefik.example.com with TLS options default" entryPointName=web
traefik-traefik-1  | time="2024-01-12T17:44:34+01:00" level=debug msg="Adding route for grafana.example.com with TLS options default" entryPointName=websecure
traefik-traefik-1  | time="2024-01-12T17:44:34+01:00" level=debug msg="Adding route for dns.example.com with TLS options default" entryPointName=websecure
traefik-traefik-1  | time="2024-01-12T17:44:34+01:00" level=debug msg="Adding route for traefik.example.com with TLS options default" entryPointName=websecure
traefik-traefik-1  | time="2024-01-12T17:44:34+01:00" level=debug msg="Trying to challenge certificate for domain [traefik.example.com] found in HostSNI rule" providerName=le.acme ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" routerName=api@docker rule="Host(`traefik.example.com`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))"
traefik-traefik-1  | time="2024-01-12T17:44:34+01:00" level=debug msg="Trying to challenge certificate for domain [traefik.example.com] found in HostSNI rule" providerName=le.acme ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" rule="Host(`traefik.example.com`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))" routerName=websecure-api@docker
traefik-traefik-1  | time="2024-01-12T17:44:34+01:00" level=debug msg="Looking for provided certificate(s) to validate [\"traefik.example.com\"]..." providerName=le.acme ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" routerName=api@docker rule="Host(`traefik.example.com`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))"
traefik-traefik-1  | time="2024-01-12T17:44:34+01:00" level=debug msg="Looking for provided certificate(s) to validate [\"traefik.example.com\"]..." routerName=websecure-api@docker providerName=le.acme ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" rule="Host(`traefik.example.com`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))"
traefik-traefik-1  | time="2024-01-12T17:44:34+01:00" level=debug msg="No ACME certificate generation required for domains [\"traefik.example.com\"]." ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" rule="Host(`traefik.example.com`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))" routerName=websecure-api@docker providerName=le.acme
traefik-traefik-1  | time="2024-01-12T17:44:34+01:00" level=debug msg="No ACME certificate generation required for domains [\"traefik.example.com\"]." rule="Host(`traefik.example.com`) && (PathPrefix(`/api`) || PathPrefix(`/dashboard`))" providerName=le.acme ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" routerName=api@docker
traefik-traefik-1  | time="2024-01-12T17:44:34+01:00" level=debug msg="Trying to challenge certificate for domain [grafana.example.com] found in HostSNI rule" rule="Host(`grafana.example.com`)" routerName=grafana@docker providerName=le.acme ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory"
traefik-traefik-1  | time="2024-01-12T17:44:34+01:00" level=debug msg="Trying to challenge certificate for domain [dns.example.com] found in HostSNI rule" ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" routerName=pihole@docker rule="Host(`dns.example.com`)" providerName=le.acme
traefik-traefik-1  | time="2024-01-12T17:44:34+01:00" level=debug msg="Looking for provided certificate(s) to validate [\"grafana.example.com\"]..." providerName=le.acme ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" rule="Host(`grafana.example.com`)" routerName=grafana@docker
traefik-traefik-1  | time="2024-01-12T17:44:34+01:00" level=debug msg="No ACME certificate generation required for domains [\"grafana.example.com\"]." providerName=le.acme ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory" rule="Host(`grafana.example.com`)" routerName=grafana@docker
traefik-traefik-1  | time="2024-01-12T17:44:34+01:00" level=debug msg="Looking for provided certificate(s) to validate [\"dns.example.com\"]..." routerName=pihole@docker rule="Host(`dns.example.com`)" providerName=le.acme ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory"
traefik-traefik-1  | time="2024-01-12T17:44:34+01:00" level=debug msg="No ACME certificate generation required for domains [\"dns.example.com\"]." routerName=pihole@docker rule="Host(`dns.example.com`)" providerName=le.acme ACME CA="https://acme-staging-v02.api.letsencrypt.org/directory"
```

</details>

### Go environment (if applicable)

<details>

```console
$ go version
go1.21.5
```

</details>


Discussion / hints:
@jankatins @AlexH-HankIT can you check that?

> The lego command is not available in the traefik docker image.

Traefik and lego are written in Go, it's a compiled language, so dependencies are not executable (it's not Python).

@TTomczek can you try lego directly? https://go-acme.github.io/lego/installation/

@ldez I tried it with the lego docker image using the following command:
```console
docker run --rm -e IONOS_API_KEY=<API_KEY> goacme/lego --accept-tos --email="me@example.com" --dns ionos --domains="*.example.com" --server="https://acme-staging-v02.api.letsencrypt.org/directory" --dns.resolvers="ns1***.ui-dns.com" run
```
After i successfully requested the certificate and waiting 45 minutes the record is still there.

Logs:
<details>

```console
2024/01/14 14:45:43 No key found for account me@example.com. Generating a P256 key.
2024/01/14 14:45:43 Saved key to /.lego/accounts/acme-staging-v02.api.letsencrypt.org/me@example.com/keys/me@example.com.key
2024/01/14 14:45:44 [INFO] acme: Registering account for me@example.com
!!!! HEADS UP !!!!

Your account credentials have been saved in your Let's Encrypt
configuration directory at "/.lego/accounts".

You should make a secure backup of this folder now. This
configuration directory will also contain certificates and
private keys obtained from Let's Encrypt so making regular
backups of this folder is ideal.
2024/01/14 14:45:44 [INFO] [*.example.com] acme: Obtaining bundled SAN certificate
2024/01/14 14:45:44 [INFO] [*.example.com] AuthURL: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/*****
2024/01/14 14:45:44 [INFO] [*.example.com] acme: use dns-01 solver
2024/01/14 14:45:44 [INFO] [*.example.com] acme: Preparing to solve DNS-01
2024/01/14 14:45:47 [INFO] [*.example.com] acme: Trying to solve DNS-01
2024/01/14 14:45:47 [INFO] [*.example.com] acme: Checking DNS record propagation using [ns1***.ui-dns.com:53]
2024/01/14 14:45:49 [INFO] Wait for propagation [timeout: 1m0s, interval: 2s]
2024/01/14 14:45:49 [INFO] [*.example.com] acme: Waiting for DNS record propagation.
2024/01/14 14:45:51 [INFO] [*.example.com] acme: Waiting for DNS record propagation.
2024/01/14 14:45:53 [INFO] [*.example.com] acme: Waiting for DNS record propagation.
2024/01/14 14:45:55 [INFO] [*.example.com] acme: Waiting for DNS record propagation.
2024/01/14 14:45:57 [INFO] [*.example.com] acme: Waiting for DNS record propagation.
2024/01/14 14:45:59 [INFO] [*.example.com] acme: Waiting for DNS record propagation.
2024/01/14 14:46:01 [INFO] [*.example.com] acme: Waiting for DNS record propagation.
2024/01/14 14:46:03 [INFO] [*.example.com] acme: Waiting for DNS record propagation.
2024/01/14 14:46:05 [INFO] [*.example.com] acme: Waiting for DNS record propagation.
2024/01/14 14:46:07 [INFO] [*.example.com] acme: Waiting for DNS record propagation.
2024/01/14 14:46:09 [INFO] [*.example.com] acme: Waiting for DNS record propagation.
2024/01/14 14:46:11 [INFO] [*.example.com] acme: Waiting for DNS record propagation.
2024/01/14 14:46:13 [INFO] [*.example.com] acme: Waiting for DNS record propagation.
2024/01/14 14:46:15 [INFO] [*.example.com] acme: Waiting for DNS record propagation.
2024/01/14 14:46:18 [INFO] [*.example.com] acme: Waiting for DNS record propagation.
2024/01/14 14:46:20 [INFO] [*.example.com] acme: Waiting for DNS record propagation.
2024/01/14 14:46:22 [INFO] [*.example.com] acme: Waiting for DNS record propagation.
2024/01/14 14:46:24 [INFO] [*.example.com] acme: Waiting for DNS record propagation.
2024/01/14 14:46:26 [INFO] [*.example.com] acme: Waiting for DNS record propagation.
2024/01/14 14:46:28 [INFO] [*.example.com] acme: Waiting for DNS record propagation.
2024/01/14 14:46:30 [INFO] [*.example.com] acme: Waiting for DNS record propagation.
2024/01/14 14:46:32 [INFO] [*.example.com] acme: Waiting for DNS record propagation.
2024/01/14 14:46:34 [INFO] [*.example.com] acme: Waiting for DNS record propagation.
2024/01/14 14:46:36 [INFO] [*.example.com] acme: Waiting for DNS record propagation.
2024/01/14 14:46:39 [INFO] [*.example.com] acme: Waiting for DNS record propagation.
2024/01/14 14:46:41 [INFO] [*.example.com] acme: Waiting for DNS record propagation.
2024/01/14 14:46:43 [INFO] [*.example.com] acme: Waiting for DNS record propagation.
2024/01/14 14:46:45 [INFO] [*.example.com] acme: Waiting for DNS record propagation.
2024/01/14 14:47:02 [INFO] [*.example.com] The server validated our request
2024/01/14 14:47:02 [INFO] [*.example.com] acme: Cleaning DNS-01 challenge
2024/01/14 14:47:05 [INFO] [*.example.com] acme: Validations succeeded; requesting certificates
2024/01/14 14:47:05 [INFO] Wait for certificate [timeout: 30s, interval: 500ms]
2024/01/14 14:47:06 [INFO] [*.example.com] Server responded with a certificate.
```

</details>
I can confirm this: I use traefik to create lets encrypt certs against my ionos hosted domain and I have a ton of 
`_acme-challenge.<subdomain>` in my TXT records for my domain.

Fix patch:
diff --git a/providers/dns/ionos/ionos.go b/providers/dns/ionos/ionos.go
index d6150a7065..2ce3de62a0 100644
--- a/providers/dns/ionos/ionos.go
+++ b/providers/dns/ionos/ionos.go
@@ -6,6 +6,7 @@ import (
 	"errors"
 	"fmt"
 	"net/http"
+	"strconv"
 	"strings"
 	"time"
 
@@ -171,8 +172,8 @@ func (d *DNSProvider) CleanUp(domain, _, keyAuth string) error {
 	}
 
 	for _, record := range records {
-		if record.Name == dns01.UnFqdn(info.EffectiveFQDN) && record.Content == info.Value {
-			err := d.client.RemoveRecord(ctx, zone.ID, record.ID)
+		if record.Name == dns01.UnFqdn(info.EffectiveFQDN) && record.Content == strconv.Quote(info.Value) {
+			err = d.client.RemoveRecord(ctx, zone.ID, record.ID)
 			if err != nil {
 				return fmt.Errorf("ionos: failed to remove record (zone=%s, record=%s): %w", zone.ID, record.ID, err)
 			}
@@ -180,7 +181,7 @@ func (d *DNSProvider) CleanUp(domain, _, keyAuth string) error {
 		}
 	}
 
-	return nil
+	return fmt.Errorf("ionos: failed to remove record, record not found (zone=%s, domain=%s, fqdn=%s, value=%s)", zone.ID, domain, info.EffectiveFQDN, info.Value)
 }
 
 func findZone(zones []internal.Zone, domain string) *internal.Zone {


Test patch: