Repository: traefik/traefik (language: go)

Issue:
Private CA Resolver and Cloudflare DNS cannot co-exist
### Welcome!

- [X] Yes, I've searched similar issues on [GitHub](https://github.com/traefik/traefik/issues) and didn't find any.
- [X] Yes, I've searched similar issues on the [Traefik community forum](https://community.traefik.io) and didn't find any.

### What did you do?

Add Cloudflare DNS resolver after running private step CA HTTP resolver

If I comment out LEGO_CA_CERTIFICATES and LEGO_CA_SERVER_NAME env vars then Cloudflare DNS works, but step CA can no longer issue certs

### What did you see instead?

```
time="2024-04-07T19:34:34Z" level=error msg="Unable to obtain ACME certificate for domains \"jellyfin.real-address.com,jf.real-address.com\": cannot get ACME client get directory at 'https://acme-v02.api.letsencrypt.org/directory': Get \"https://acme-v02.api.letsencrypt.org/directory\": tls: failed to verify certificate: x509: certificate is valid for acme-v02-1.api.letsencrypt.org, acme-v02-2.api.letsencrypt.org, acme-v02-3.api.letsencrypt.org, acme-v02-4.api.letsencrypt.org, acme-v02-5.api.letsencrypt.org, acme-v02.api.letsencrypt.org, incident.letsencrypt.org, not step-ca" ACME CA="https://acme-v02.api.letsencrypt.org/directory" rule="Host(`jellyfin.real-address.com`, `jf.real-address.com`)" routerName=jellyfin-real-address-https@docker providerName=cloudflare.acme
```

### What version of Traefik are you using?

v2.11.0

### What is your environment & configuration?

```yaml
## Jellyfin Labels

x-nio_io_labels: &nio_io_labels
  ## HTTPS Service
  traefik.http.routers.jellyfin-https.service: jellyfin-https
  traefik.http.routers.jellyfin-https.rule: >-
    Host(`jellyfin-${DEVICE_IP:-ip_addr}.nip.io`, `jf-${DEVICE_IP:-ip_addr}.nip.io`, `jellyfin.nip.io`, `jf.nip.io`)
  traefik.http.routers.jellyfin-https.entrypoints: websecure
  traefik.http.routers.jellyfin-https.middlewares: jellyfin-mw
  traefik.http.routers.jellyfin-https.tls: true
  traefik.http.routers.jellyfin-https.tls.certresolver: step-ca
  traefik.http.services.jellyfin-https.loadbalancer.server.port: 8096
  # HTTP Service Allowed!
  traefik.http.routers.jellyfin-http.service: jellyfin-http
  traefik.http.routers.jellyfin-http.rule: >-
    Host(`jellyfin-${DEVICE_IP:-ip_addr}.nip.io`, `jf-${DEVICE_IP:-ip_addr}.nip.io`, `jellyfin.nip.io`, `jf.nip.io`)
  traefik.http.routers.jellyfin-http.entrypoints: web
  traefik.http.routers.jellyfin-http.middlewares: jellyfin-mw
  traefik.http.services.jellyfin-http.loadbalancer.server.port: 8096

## UDP

x-udp_labels: &udp_labels
  traefik.udp.routers.jellyfin-client-discovery.entrypoints: client_discovery
  traefik.udp.routers.jellyfin-client-discovery.service: jellyfin-client-discovery
  traefik.udp.services.jellyfin-client-discovery.loadbalancer.server.port: 7359

## Recommended by Jellyfin Docs

x-recommended_middleware_labels: &recommended_middleware_labels
  traefik.http.middlewares.jellyfin-mw.headers.customResponseHeaders.X-Robots-Tag: noindex,nofollow,nosnippet,noarchive,notranslate,noimageindex
  traefik.http.middlewares.jellyfin-mw.headers.frameDeny: 'true'
  traefik.http.middlewares.jellyfin-mw.headers.contentTypeNosniff: 'true'
  traefik.http.middlewares.jellyfin-mw.headers.browserXSSFilter: 'true'
  traefik.http.services.jellyfin.loadBalancer.passHostHeader: 'true'

x-real_address_labels: &real_address_labels
  # HTTPS Service
  traefik.http.routers.jellyfin-real-address-https.service: jellyfin-real-address-https
  traefik.http.routers.jellyfin-real-address-https.rule: Host(`jellyfin.real-address.com`, `jf.real-address.com`)
  traefik.http.routers.jellyfin-real-address-https.entrypoints: websecure
  traefik.http.routers.jellyfin-real-address-https.middlewares: jellyfin-mw
  traefik.http.routers.jellyfin-real-address-https.tls: true
  traefik.http.routers.jellyfin-real-address-https.tls.certresolver: cloudflare
  traefik.http.services.jellyfin-real-address-https.loadbalancer.server.port: 8096
  # HTTP to HTTPS Redirect
  traefik.http.routers.jellyfin-real-address-http.rule: Host(`jellyfin.real-address.com`, `jf.real-address.com`)
  traefik.http.routers.jellyfin-real-address-http.entrypoints: web
  traefik.http.routers.jellyfin-real-address-http.middlewares: https_redirect
```

Traefik ENV Vars
```yaml
services:
  traefik:
    image: traefik:latest
    container_name: traefik
    environment:
      # LEGO_CA_CERTIFICATES: /shared_files/ca_certs/root_ca.crt
      # LEGO_CA_SERVER_NAME: step-ca
      SSL_CERT_FILE: /shared_files/ca_certs/root_ca.crt
      CLOUDFLARE_DNS_API_TOKEN: $CLOUDFLARE_DNS_API_TOKEN
```
Add more configuration information here.
```yaml
# Static Conf
# Configure your CertificateResolver here...
# ---
certificatesResolvers:
  step-ca:
    acme:
      email: redacted@gmail.com
      storage: /shared_files/acme/acme.json
      caServer: "https://192.168.1.66:55888/acme/acme/directory"
      # caServer: "https://step-ca:55888/acme/acme/directory"
      httpChallenge:
        entryPoint: web
  cloudflare:
    acme:
      email: redacted@gmail.com
      storage: /shared_files/acme/cloudflare_dns_acme.json
      caServer: "https://acme-v02.api.letsencrypt.org/directory"
      dnsChallenge:
        provider: cloudflare
        delayBeforeCheck: 10
```

### If applicable, please paste the log output in DEBUG level

_No response_


Discussion / hints:
Hello,

can you try to set the env var `LEGO_CA_SYSTEM_CERT_POOL` to `true`?

Hi thanks for responding, I left it out of what I pasted but I tried that variable as well, no difference. I will try it again and let you know.
Another way is to add the Step-ca root CA to the system-wide trusted root list.

Doesn't that involve making a new image?
I don't try it but I think you can just mount the CA file.

Fix patch:
diff --git a/docs/content/https/acme.md b/docs/content/https/acme.md
index b73e6cc2f2..daaaa21000 100644
--- a/docs/content/https/acme.md
+++ b/docs/content/https/acme.md
@@ -709,6 +709,109 @@ certificatesResolvers:
 # ...
 ```
 
+### `caCertificates`
+
+_Optional, Default=[]_
+
+The `caCertificates` option specifies the paths to PEM encoded CA Certificates that can be used to authenticate an ACME server with an HTTPS certificate not issued by a CA in the system-wide trusted root list.
+
+```yaml tab="File (YAML)"
+certificatesResolvers:
+  myresolver:
+    acme:
+      # ...
+      caCertificates:
+        - path/certificates1.pem
+        - path/certificates2.pem
+      # ...
+```
+
+```toml tab="File (TOML)"
+[certificatesResolvers.myresolver.acme]
+  # ...
+  caCertificates = [ "path/certificates1.pem", "path/certificates2.pem" ]
+  # ...
+```
+
+```bash tab="CLI"
+# ...
+--certificatesresolvers.myresolver.acme.caCertificates="path/certificates1.pem,path/certificates2.pem"
+# ...
+```
+
+??? note "LEGO Environment Variable"
+
+    It can be defined globally by using the environment variable `LEGO_CA_CERTIFICATES`.
+    This environment variable is neither a fallback nor an override of the configuration option.
+
+### `caSystemCertPool`
+
+_Optional, Default=false_
+
+The `caSystemCertPool` option defines if the certificates pool must use a copy of the system cert pool.
+
+```yaml tab="File (YAML)"
+certificatesResolvers:
+  myresolver:
+    acme:
+      # ...
+      caSystemCertPool: true
+      # ...
+```
+
+```toml tab="File (TOML)"
+[certificatesResolvers.myresolver.acme]
+  # ...
+  caSystemCertPool = true
+  # ...
+```
+
+```bash tab="CLI"
+# ...
+--certificatesresolvers.myresolver.acme.caSystemCertPool=true
+# ...
+```
+
+??? note "LEGO Environment Variable"
+
+    It can be defined globally by using the environment variable `LEGO_CA_SYSTEM_CERT_POOL`.
+    `LEGO_CA_SYSTEM_CERT_POOL` is ignored if `LEGO_CA_CERTIFICATES` is not set or empty.
+    This environment variable is neither a fallback nor an override of the configuration option.
+
+### `caServerName`
+
+_Optional, Default=""_
+
+The `caServerName` option specifies the CA server name that can be used to authenticate an ACME server with an HTTPS certificate not issued by a CA in the system-wide trusted root list.
+
+```yaml tab="File (YAML)"
+certificatesResolvers:
+  myresolver:
+    acme:
+      # ...
+      caServerName: "my-server"
+      # ...
+```
+
+```toml tab="File (TOML)"
+[certificatesResolvers.myresolver.acme]
+  # ...
+  caServerName = "my-server"
+  # ...
+```
+
+```bash tab="CLI"
+# ...
+--certificatesresolvers.myresolver.acme.caServerName="my-server"
+# ...
+```
+
+??? note "LEGO Environment Variable"
+
+    It can be defined globally by using the environment variable `LEGO_CA_SERVER_NAME`.
+    `LEGO_CA_SERVER_NAME` is ignored if `LEGO_CA_CERTIFICATES` is not set or empty.
+    This environment variable is neither a fallback nor an override of the configuration option.
+
 ## Fallback
 
 If Let's Encrypt is not reachable, the following certificates will apply:
diff --git a/docs/content/reference/static-configuration/cli-ref.md b/docs/content/reference/static-configuration/cli-ref.md
index a2171dfa8d..b0c02c5653 100644
--- a/docs/content/reference/static-configuration/cli-ref.md
+++ b/docs/content/reference/static-configuration/cli-ref.md
@@ -57,9 +57,18 @@ Activate API directly on the entryPoint named traefik. (Default: ```false```)
 `--certificatesresolvers.<name>`:  
 Certificates resolvers configuration. (Default: ```false```)
 
+`--certificatesresolvers.<name>.acme.cacertificates`:  
+Specify the paths to PEM encoded CA Certificates that can be used to authenticate an ACME server with an HTTPS certificate not issued by a CA in the system-wide trusted root list.
+
 `--certificatesresolvers.<name>.acme.caserver`:  
 CA server to use. (Default: ```https://acme-v02.api.letsencrypt.org/directory```)
 
+`--certificatesresolvers.<name>.acme.caservername`:  
+Specify the CA server name that can be used to authenticate an ACME server with an HTTPS certificate not issued by a CA in the system-wide trusted root list.
+
+`--certificatesresolvers.<name>.acme.casystemcertpool`:  
+Define if the certificates pool must use a copy of the system cert pool. (Default: ```false```)
+
 `--certificatesresolvers.<name>.acme.certificatesduration`:  
 Certificates' duration in hours. (Default: ```2160```)
 
diff --git a/docs/content/reference/static-configuration/env-ref.md b/docs/content/reference/static-configuration/env-ref.md
index b7cd5ee3c5..2f4c19971a 100644
--- a/docs/content/reference/static-configuration/env-ref.md
+++ b/docs/content/reference/static-configuration/env-ref.md
@@ -57,9 +57,18 @@ Activate API directly on the entryPoint named traefik. (Default: ```false```)
 `TRAEFIK_CERTIFICATESRESOLVERS_<NAME>`:  
 Certificates resolvers configuration. (Default: ```false```)
 
+`TRAEFIK_CERTIFICATESRESOLVERS_<NAME>_ACME_CACERTIFICATES`:  
+Specify the paths to PEM encoded CA Certificates that can be used to authenticate an ACME server with an HTTPS certificate not issued by a CA in the system-wide trusted root list.
+
 `TRAEFIK_CERTIFICATESRESOLVERS_<NAME>_ACME_CASERVER`:  
 CA server to use. (Default: ```https://acme-v02.api.letsencrypt.org/directory```)
 
+`TRAEFIK_CERTIFICATESRESOLVERS_<NAME>_ACME_CASERVERNAME`:  
+Specify the CA server name that can be used to authenticate an ACME server with an HTTPS certificate not issued by a CA in the system-wide trusted root list.
+
+`TRAEFIK_CERTIFICATESRESOLVERS_<NAME>_ACME_CASYSTEMCERTPOOL`:  
+Define if the certificates pool must use a copy of the system cert pool. (Default: ```false```)
+
 `TRAEFIK_CERTIFICATESRESOLVERS_<NAME>_ACME_CERTIFICATESDURATION`:  
 Certificates' duration in hours. (Default: ```2160```)
 
diff --git a/docs/content/reference/static-configuration/file.toml b/docs/content/reference/static-configuration/file.toml
index daccb6dee1..f92cd37631 100644
--- a/docs/content/reference/static-configuration/file.toml
+++ b/docs/content/reference/static-configuration/file.toml
@@ -441,6 +441,9 @@
       storage = "foobar"
       keyType = "foobar"
       certificatesDuration = 42
+      caCertificates = ["foobar", "foobar"]
+      caSystemCertPool = true
+      caServerName = "foobar"
       [certificatesResolvers.CertificateResolver0.acme.eab]
         kid = "foobar"
         hmacEncoded = "foobar"
@@ -461,6 +464,9 @@
       storage = "foobar"
       keyType = "foobar"
       certificatesDuration = 42
+      caCertificates = ["foobar", "foobar"]
+      caSystemCertPool = true
+      caServerName = "foobar"
       [certificatesResolvers.CertificateResolver1.acme.eab]
         kid = "foobar"
         hmacEncoded = "foobar"
diff --git a/docs/content/reference/static-configuration/file.yaml b/docs/content/reference/static-configuration/file.yaml
index 2919824c3a..12b5efccc0 100644
--- a/docs/content/reference/static-configuration/file.yaml
+++ b/docs/content/reference/static-configuration/file.yaml
@@ -483,6 +483,11 @@ certificatesResolvers:
         kid: foobar
         hmacEncoded: foobar
       certificatesDuration: 42
+      caCertificates:
+        - foobar
+        - foobar
+      caSystemCertPool: true
+      caServerName: foobar
       dnsChallenge:
         provider: foobar
         delayBeforeCheck: 42s
@@ -505,6 +510,11 @@ certificatesResolvers:
         kid: foobar
         hmacEncoded: foobar
       certificatesDuration: 42
+      caCertificates:
+        - foobar
+        - foobar
+      caSystemCertPool: true
+      caServerName: foobar
       dnsChallenge:
         provider: foobar
         delayBeforeCheck: 42s
diff --git a/pkg/provider/acme/provider.go b/pkg/provider/acme/provider.go
index 7df870c595..912b4073eb 100644
--- a/pkg/provider/acme/provider.go
+++ b/pkg/provider/acme/provider.go
@@ -6,9 +6,13 @@ import (
 	"crypto/x509"
 	"errors"
 	"fmt"
+	"net"
+	"net/http"
 	"net/url"
+	"os"
 	"reflect"
 	"sort"
+	"strconv"
 	"strings"
 	"sync"
 	"time"
@@ -43,6 +47,10 @@ type Configuration struct {
 	EAB                  *EAB   `description:"External Account Binding to use." json:"eab,omitempty" toml:"eab,omitempty" yaml:"eab,omitempty"`
 	CertificatesDuration int    `description:"Certificates' duration in hours." json:"certificatesDuration,omitempty" toml:"certificatesDuration,omitempty" yaml:"certificatesDuration,omitempty" export:"true"`
 
+	CACertificates   []string `description:"Specify the paths to PEM encoded CA Certificates that can be used to authenticate an ACME server with an HTTPS certificate not issued by a CA in the system-wide trusted root list." json:"caCertificates,omitempty" toml:"caCertificates,omitempty" yaml:"caCertificates,omitempty"`
+	CASystemCertPool bool     `description:"Define if the certificates pool must use a copy of the system cert pool." json:"caSystemCertPool,omitempty" toml:"caSystemCertPool,omitempty" yaml:"caSystemCertPool,omitempty" export:"true"`
+	CAServerName     string   `description:"Specify the CA server name that can be used to authenticate an ACME server with an HTTPS certificate not issued by a CA in the system-wide trusted root list." json:"caServerName,omitempty" toml:"caServerName,omitempty" yaml:"caServerName,omitempty" export:"true"`
+
 	DNSChallenge  *DNSChallenge  `description:"Activate DNS-01 Challenge." json:"dnsChallenge,omitempty" toml:"dnsChallenge,omitempty" yaml:"dnsChallenge,omitempty" label:"allowEmpty" file:"allowEmpty" export:"true"`
 	HTTPChallenge *HTTPChallenge `description:"Activate HTTP-01 Challenge." json:"httpChallenge,omitempty" toml:"httpChallenge,omitempty" yaml:"httpChallenge,omitempty" label:"allowEmpty" file:"allowEmpty" export:"true"`
 	TLSChallenge  *TLSChallenge  `description:"Activate TLS-ALPN-01 Challenge." json:"tlsChallenge,omitempty" toml:"tlsChallenge,omitempty" yaml:"tlsChallenge,omitempty" label:"allowEmpty" file:"allowEmpty" export:"true"`
@@ -261,6 +269,11 @@ func (p *Provider) getClient() (*lego.Client, error) {
 	config.Certificate.KeyType = GetKeyType(ctx, p.KeyType)
 	config.UserAgent = fmt.Sprintf("containous-traefik/%s", version.Version)
 
+	config.HTTPClient, err = p.createHTTPClient()
+	if err != nil {
+		return nil, fmt.Errorf("creating HTTP client: %w", err)
+	}
+
 	client, err := lego.NewClient(config)
 	if err != nil {
 		return nil, err
@@ -340,6 +353,64 @@ func (p *Provider) getClient() (*lego.Client, error) {
 	return p.client, nil
 }
 
+func (p *Provider) createHTTPClient() (*http.Client, error) {
+	tlsConfig, err := p.createClientTLSConfig()
+	if err != nil {
+		return nil, fmt.Errorf("creating client TLS config: %w", err)
+	}
+
+	return &http.Client{
+		Timeout: 2 * time.Minute,
+		Transport: &http.Transport{
+			Proxy: http.ProxyFromEnvironment,
+			DialContext: (&net.Dialer{
+				Timeout:   30 * time.Second,
+				KeepAlive: 30 * time.Second,
+			}).DialContext,
+			TLSHandshakeTimeout:   30 * time.Second,
+			ResponseHeaderTimeout: 30 * time.Second,
+			TLSClientConfig:       tlsConfig,
+		},
+	}, nil
+}
+
+func (p *Provider) createClientTLSConfig() (*tls.Config, error) {
+	if len(p.CACertificates) > 0 || p.CAServerName != "" {
+		certPool, err := lego.CreateCertPool(p.CACertificates, p.CASystemCertPool)
+		if err != nil {
+			return nil, fmt.Errorf("creating cert pool with custom certificates: %w", err)
+		}
+
+		return &tls.Config{
+			ServerName: p.CAServerName,
+			RootCAs:    certPool,
+		}, nil
+	}
+
+	// Compatibility layer with the lego.
+	// https://github.com/go-acme/lego/blob/834a9089f143e3407b3f5c8b93a0e285ba231fe2/lego/client_config.go#L24-L34
+	// https://github.com/go-acme/lego/blob/834a9089f143e3407b3f5c8b93a0e285ba231fe2/lego/client_config.go#L97-L113
+
+	serverName := os.Getenv("LEGO_CA_SERVER_NAME")
+	customCACertsPath := os.Getenv("LEGO_CA_CERTIFICATES")
+
+	if customCACertsPath == "" && serverName == "" {
+		return nil, nil
+	}
+
+	useSystemCertPool, _ := strconv.ParseBool(os.Getenv("LEGO_CA_SYSTEM_CERT_POOL"))
+
+	certPool, err := lego.CreateCertPool(strings.Split(customCACertsPath, string(os.PathListSeparator)), useSystemCertPool)
+	if err != nil {
+		return nil, fmt.Errorf("creating cert pool: %w", err)
+	}
+
+	return &tls.Config{
+		ServerName: serverName,
+		RootCAs:    certPool,
+	}, nil
+}
+
 func (p *Provider) initAccount(ctx context.Context) (*Account, error) {
 	if p.account == nil || len(p.account.Email) == 0 {
 		var err error
@@ -424,8 +495,7 @@ func (p *Provider) watchNewDomains(ctx context.Context) {
 
 						if len(route.TLS.Domains) > 0 {
 							domains := deleteUnnecessaryDomains(ctxRouter, route.TLS.Domains)
-							for i := range len(domains) {
-								domain := domains[i]
+							for _, domain := range domains {
 								safe.Go(func() {
 									dom, cert, err := p.resolveCertificate(ctx, domain, traefiktls.DefaultTLSStoreName)
 									if err != nil {
@@ -461,8 +531,7 @@ func (p *Provider) watchNewDomains(ctx context.Context) {
 
 						if len(route.TLS.Domains) > 0 {
 							domains := deleteUnnecessaryDomains(ctxRouter, route.TLS.Domains)
-							for i := range len(domains) {
-								domain := domains[i]
+							for _, domain := range domains {
 								safe.Go(func() {
 									dom, cert, err := p.resolveCertificate(ctx, domain, traefiktls.DefaultTLSStoreName)
 									if err != nil {


Test patch: