Repository: caddyserver/caddy (language: go)

Issue:
"get_certificate tailscale" forces "on_demand_tls" global option
**Backstory**: I have a docker/caddy setup where I use a docker volume to expose the tailscaled socket to Caddy. It randomly stopped working today, so I tried to fix it. It looked like there were recent changes with socket paths so I tried different combinations with /tmp and /run but nothing worked. (EDIT: the error for this is `caddy_vaultwarden | {"level":"debug","ts":1706154566.203921,"logger":"http.stdlib","msg":"http: TLS handshake error from 100.107.69.109:41312: no certificate available for 'redacted.x.ts.net'"}`)

**The problem**: In order to debug this I tried enabling `get_certificate tailscale` under the tls directive, but then it starts enforcing the "ask" endpoint which doesn't make sense for tailscale users. An example of the error is [here](https://caddy.community/t/on-demand-fails-suddenly-in-2-7-4/21320).

I believe the "ask" enforcing is a fairly recent change, and so this may have been overlooked when the change was implemented. I know the docker volume approach isn't officially supported but it'd be great if this issue could be resolved so that I can try and debug it myself. Thanks!


Discussion / hints:
What version are you using? What's the error message you see, exactly? Show your logs & config.
Yep, my bad, I was in a rush yesterday. Here's the info:
Version: `caddy:latest` on docker (v2.7.6)

Caddyfile:
```
{
    debug
}

:443 {
  log {
    level INFO
    output file {$LOG_FILE} {
      roll_size 10MB
      roll_keep 10
    }
  }
  tls {
    get_certificate tailscale # Didn't need this before (ever since the feature was released)
  }

   encode gzip
    
   reverse_proxy vaultwarden:80
   reverse_proxy /notifications/hub vaultwarden:3012
}
```
Error:
```
caddy_vaultwarden | {"level":"info","ts":1706153302.1186092,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
caddy_vaultwarden | {"level":"warn","ts":1706153302.1194217,"msg":"Caddyfile input is not formatted; run 'caddy fmt --overwrite' to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":2}
caddy_vaultwarden | {"level":"info","ts":1706153302.1199346,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//[::1]:2019","//127.0.0.1:2019","//localhost:2019"]}
caddy_vaultwarden | {"level":"info","ts":1706153302.1200917,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc0006f5880"}
caddy_vaultwarden | {"level":"info","ts":1706153302.1201015,"logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0xc0006f5880"}
caddy_vaultwarden | Error: loading initial config: loading new config: loading http app module: provision http: getting tls app: loading tls app module: provision tls: provisioning automation policy 0: on-demand TLS cannot be enabled without an 'ask' endpoint to prevent abuse; please refer to documentation for details
caddy_vaultwarden | {"level":"info","ts":1706153303.7275922,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
caddy_vaultwarden | {"level":"warn","ts":1706153303.7283955,"msg":"Caddyfile input is not formatted; run 'caddy fmt --overwrite' to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":2}
caddy_vaultwarden | {"level":"info","ts":1706153303.7289453,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
caddy_vaultwarden | {"level":"info","ts":1706153303.729059,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc000789300"}
caddy_vaultwarden | {"level":"info","ts":1706153303.729068,"logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0xc000789300"}
caddy_vaultwarden | Error: loading initial config: loading new config: loading http app module: provision http: getting tls app: loading tls app module: provision tls: provisioning automation policy 0: on-demand TLS cannot be enabled without an 'ask' endpoint to prevent abuse; please refer to documentation for details
caddy_vaultwarden exited with code 1
```

EDIT: I have managed to work around this using the solution given in #5986. 
Hi!

Just want to report the same issue as it is still open, but using `get_certificate http` and not `tailscale`.

Previously I used `tls cert key` with a separate cron job fetching fresh wildcard cert and reloading Caddy at a given interval before trying out a new config using `get_certificate`. After config rewrite and trying to reload it fails with the same error message as above.

It works with the work-around as mentioned in #5986. 

Providers cert endpoint returns `200 OK` as expected.

**Version**
`v2.7.6 h1:w0NymbG2m9PcvKWsrXO6EEkY9Ru4FJK8uQbYcev1p3A=`
Running as service on Ubuntu 22.04 LTS.

**Config**
```
{
        debug
}

*.mydomain.com {
        tls {
                # Extra query params as specified by provider
                get_certificate http https://cert.myprovider.com?domain=*.mydomain.com&secret=SECRET
        }

        log {
                format json
                output file /var/log/caddy/wildcard.log
        }

        @app1 host app1.mydomain.com
        handle @app1 {
                reverse_proxy localhost:3000
        }

        respond "No service running on {host}, yet...."
}
``` 
**Log on `caddy reload`**
```json
{
  "level": "error",
  "ts": 1712191729.55823,
  "logger": "admin.api",
  "msg": "request error",
  "error": "loading config: loading new config: loading http app module: provision http: getting tls app: loading tls app module: provision tls: provisioning automation policy 1: on-demand TLS cannot be enabled without an 'ask' endpoint to prevent abuse; please refer to documentation for details",
  "status_code": 400
}
``` 

Thanks for the great work you are doing with Caddy 👏 

Fix patch:
diff --git a/admin.go b/admin.go
index 13d24084dab..3950a92facc 100644
--- a/admin.go
+++ b/admin.go
@@ -474,7 +474,6 @@ func manageIdentity(ctx Context, cfg *Config) error {
 	// import the caddytls package -- but it works
 	if cfg.Admin.Identity.IssuersRaw == nil {
 		cfg.Admin.Identity.IssuersRaw = []json.RawMessage{
-			json.RawMessage(`{"module": "zerossl"}`),
 			json.RawMessage(`{"module": "acme"}`),
 		}
 	}
diff --git a/caddyconfig/httpcaddyfile/builtins.go b/caddyconfig/httpcaddyfile/builtins.go
index 505885d2d51..a7f4247f57d 100644
--- a/caddyconfig/httpcaddyfile/builtins.go
+++ b/caddyconfig/httpcaddyfile/builtins.go
@@ -24,7 +24,7 @@ import (
 	"time"
 
 	"github.com/caddyserver/certmagic"
-	"github.com/mholt/acmez/acme"
+	"github.com/mholt/acmez/v2/acme"
 	"go.uber.org/zap/zapcore"
 
 	"github.com/caddyserver/caddy/v2"
@@ -107,7 +107,6 @@ func parseTLS(h Helper) ([]ConfigValue, error) {
 	var onDemand bool
 	var reusePrivateKeys bool
 
-	// file certificate loader
 	firstLine := h.RemainingArgs()
 	switch len(firstLine) {
 	case 0:
@@ -117,13 +116,13 @@ func parseTLS(h Helper) ([]ConfigValue, error) {
 		} else if !strings.Contains(firstLine[0], "@") {
 			return nil, h.Err("single argument must either be 'internal' or an email address")
 		} else {
-			if acmeIssuer == nil {
-				acmeIssuer = new(caddytls.ACMEIssuer)
+			acmeIssuer = &caddytls.ACMEIssuer{
+				Email: firstLine[0],
 			}
-			acmeIssuer.Email = firstLine[0]
 		}
 
 	case 2:
+		// file certificate loader
 		certFilename := firstLine[0]
 		keyFilename := firstLine[1]
 
@@ -488,19 +487,24 @@ func parseTLS(h Helper) ([]ConfigValue, error) {
 
 	case acmeIssuer != nil:
 		// implicit ACME issuers (from various subdirectives) - use defaults; there might be more than one
-		defaultIssuers := caddytls.DefaultIssuers()
+		defaultIssuers := caddytls.DefaultIssuers(acmeIssuer.Email)
 
-		// if a CA endpoint was set, override multiple implicit issuers since it's a specific one
+		// if an ACME CA endpoint was set, the user expects to use that specific one,
+		// not any others that may be defaults, so replace all defaults with that ACME CA
 		if acmeIssuer.CA != "" {
 			defaultIssuers = []certmagic.Issuer{acmeIssuer}
 		}
 
 		for _, issuer := range defaultIssuers {
-			switch iss := issuer.(type) {
-			case *caddytls.ACMEIssuer:
-				issuer = acmeIssuer
-			case *caddytls.ZeroSSLIssuer:
-				iss.ACMEIssuer = acmeIssuer
+			// apply settings from the implicitly-configured ACMEIssuer to any
+			// default ACMEIssuers, but preserve each default issuer's CA endpoint,
+			// because, for example, if you configure the DNS challenge, it should
+			// apply to any of the default ACMEIssuers, but you don't want to trample
+			// out their unique CA endpoints
+			if iss, ok := issuer.(*caddytls.ACMEIssuer); ok && iss != nil {
+				acmeCopy := *acmeIssuer
+				acmeCopy.CA = iss.CA
+				issuer = &acmeCopy
 			}
 			configVals = append(configVals, ConfigValue{
 				Class: "tls.cert_issuer",
diff --git a/caddyconfig/httpcaddyfile/options.go b/caddyconfig/httpcaddyfile/options.go
index 70d475d6b28..bbc63ced8e0 100644
--- a/caddyconfig/httpcaddyfile/options.go
+++ b/caddyconfig/httpcaddyfile/options.go
@@ -18,7 +18,7 @@ import (
 	"strconv"
 
 	"github.com/caddyserver/certmagic"
-	"github.com/mholt/acmez/acme"
+	"github.com/mholt/acmez/v2/acme"
 
 	"github.com/caddyserver/caddy/v2"
 	"github.com/caddyserver/caddy/v2/caddyconfig"
@@ -212,9 +212,9 @@ func parseOptACMEDNS(d *caddyfile.Dispenser, _ any) (any, error) {
 	if err != nil {
 		return nil, err
 	}
-	prov, ok := unm.(certmagic.ACMEDNSProvider)
+	prov, ok := unm.(certmagic.DNSProvider)
 	if !ok {
-		return nil, d.Errf("module %s (%T) is not a certmagic.ACMEDNSProvider", modID, unm)
+		return nil, d.Errf("module %s (%T) is not a certmagic.DNSProvider", modID, unm)
 	}
 	return prov, nil
 }
diff --git a/caddyconfig/httpcaddyfile/tlsapp.go b/caddyconfig/httpcaddyfile/tlsapp.go
index 1adb2b6e027..08da3a5c7f3 100644
--- a/caddyconfig/httpcaddyfile/tlsapp.go
+++ b/caddyconfig/httpcaddyfile/tlsapp.go
@@ -24,7 +24,7 @@ import (
 	"strings"
 
 	"github.com/caddyserver/certmagic"
-	"github.com/mholt/acmez/acme"
+	"github.com/mholt/acmez/v2/acme"
 
 	"github.com/caddyserver/caddy/v2"
 	"github.com/caddyserver/caddy/v2/caddyconfig"
@@ -224,7 +224,7 @@ func (st ServerType) buildTLSApp(
 				var internal, external []string
 				for _, s := range ap.SubjectsRaw {
 					// do not create Issuers for Tailscale domains; they will be given a Manager instead
-					if strings.HasSuffix(strings.ToLower(s), ".ts.net") {
+					if isTailscaleDomain(s) {
 						continue
 					}
 					if !certmagic.SubjectQualifiesForCert(s) {
@@ -378,15 +378,12 @@ func (st ServerType) buildTLSApp(
 				if len(ap.Issuers) == 0 && automationPolicyHasAllPublicNames(ap) {
 					// for public names, create default issuers which will later be filled in with configured global defaults
 					// (internal names will implicitly use the internal issuer at auto-https time)
-					ap.Issuers = caddytls.DefaultIssuers()
+					emailStr, _ := globalEmail.(string)
+					ap.Issuers = caddytls.DefaultIssuers(emailStr)
 
 					// if a specific endpoint is configured, can't use multiple default issuers
 					if globalACMECA != nil {
-						if strings.Contains(globalACMECA.(string), "zerossl") {
-							ap.Issuers = []certmagic.Issuer{&caddytls.ZeroSSLIssuer{ACMEIssuer: new(caddytls.ACMEIssuer)}}
-						} else {
-							ap.Issuers = []certmagic.Issuer{new(caddytls.ACMEIssuer)}
-						}
+						ap.Issuers = []certmagic.Issuer{new(caddytls.ACMEIssuer)}
 					}
 				}
 			}
@@ -666,17 +663,33 @@ func automationPolicyShadows(i int, aps []*caddytls.AutomationPolicy) int {
 // subjectQualifiesForPublicCert is like certmagic.SubjectQualifiesForPublicCert() except
 // that this allows domains with multiple wildcard levels like '*.*.example.com' to qualify
 // if the automation policy has OnDemand enabled (i.e. this function is more lenient).
+//
+// IP subjects are considered as non-qualifying for public certs. Technically, there are
+// now public ACME CAs as well as non-ACME CAs that issue IP certificates. But this function
+// is used solely for implicit automation (defaults), where it gets really complicated to
+// keep track of which issuers support IP certificates in which circumstances. Currently,
+// issuers that support IP certificates are very few, and all require some sort of config
+// from the user anyway (such as an account credential). Since we cannot implicitly and
+// automatically get public IP certs without configuration from the user, we treat IPs as
+// not qualifying for public certificates. Users should expressly configure an issuer
+// that supports IP certs for that purpose.
 func subjectQualifiesForPublicCert(ap *caddytls.AutomationPolicy, subj string) bool {
 	return !certmagic.SubjectIsIP(subj) &&
 		!certmagic.SubjectIsInternal(subj) &&
 		(strings.Count(subj, "*.") < 2 || ap.OnDemand)
 }
 
+// automationPolicyHasAllPublicNames returns true if all the names on the policy
+// do NOT qualify for public certs OR are tailscale domains.
 func automationPolicyHasAllPublicNames(ap *caddytls.AutomationPolicy) bool {
 	for _, subj := range ap.SubjectsRaw {
-		if !subjectQualifiesForPublicCert(ap, subj) {
+		if !subjectQualifiesForPublicCert(ap, subj) || isTailscaleDomain(subj) {
 			return false
 		}
 	}
 	return true
 }
+
+func isTailscaleDomain(name string) bool {
+	return strings.HasSuffix(strings.ToLower(name), ".ts.net")
+}
diff --git a/cmd/caddy/setcap.sh b/cmd/caddy/setcap.sh
index ac5335c5601..39aea2d6087 100755
--- a/cmd/caddy/setcap.sh
+++ b/cmd/caddy/setcap.sh
@@ -1,13 +1,16 @@
 #!/bin/sh
 
-# USAGE: go run -exec ./setcap.sh main.go <args...>
+# USAGE:
+# 	go run -exec ./setcap.sh main.go <args...>
 #
 # (Example: `go run -exec ./setcap.sh main.go run --config caddy.json`)
 #
 # For some reason this does not work on my Arch system, so if you find that's
-# the case, you can instead do: go build && ./setcap.sh ./caddy <args...>
-# but this will leave the ./caddy binary laying around.
+# the case, you can instead do:
+#
+# 	go build && ./setcap.sh ./caddy <args...>
 #
+# but this will leave the ./caddy binary laying around.
 #
 
 sudo setcap cap_net_bind_service=+ep "$1"
diff --git a/go.mod b/go.mod
index e4de7d29bfa..c3dc1568ff6 100644
--- a/go.mod
+++ b/go.mod
@@ -1,22 +1,21 @@
 module github.com/caddyserver/caddy/v2
 
-go 1.21
-
-toolchain go1.21.4
+go 1.22.0
 
 require (
 	github.com/BurntSushi/toml v1.3.2
 	github.com/Masterminds/sprig/v3 v3.2.3
 	github.com/alecthomas/chroma/v2 v2.13.0
 	github.com/aryann/difflib v0.0.0-20210328193216-ff5ff6dc229b
-	github.com/caddyserver/certmagic v0.20.0
+	github.com/caddyserver/certmagic v0.20.1-0.20240412214119-167015dd6570
+	github.com/caddyserver/zerossl v0.1.2
 	github.com/dustin/go-humanize v1.0.1
 	github.com/go-chi/chi/v5 v5.0.12
 	github.com/google/cel-go v0.20.0
 	github.com/google/uuid v1.6.0
 	github.com/klauspost/compress v1.17.0
-	github.com/klauspost/cpuid/v2 v2.2.5
-	github.com/mholt/acmez v1.2.0
+	github.com/klauspost/cpuid/v2 v2.2.7
+	github.com/mholt/acmez/v2 v2.0.0-beta.2
 	github.com/prometheus/client_golang v1.19.0
 	github.com/quic-go/quic-go v0.42.0
 	github.com/smallstep/certificates v0.25.3-rc5
@@ -112,12 +111,12 @@ require (
 	github.com/jackc/pgservicefile v0.0.0-20221227161230-091c0ba34f0a // indirect
 	github.com/jackc/pgtype v1.14.0 // indirect
 	github.com/jackc/pgx/v4 v4.18.3 // indirect
-	github.com/libdns/libdns v0.2.1 // indirect
+	github.com/libdns/libdns v0.2.2 // indirect
 	github.com/manifoldco/promptui v0.9.0 // indirect
 	github.com/mattn/go-colorable v0.1.13 // indirect
 	github.com/mattn/go-isatty v0.0.20 // indirect
 	github.com/mgutz/ansi v0.0.0-20200706080929-d51e80ef957d // indirect
-	github.com/miekg/dns v1.1.55 // indirect
+	github.com/miekg/dns v1.1.58 // indirect
 	github.com/mitchellh/copystructure v1.2.0 // indirect
 	github.com/mitchellh/go-ps v1.0.0 // indirect
 	github.com/mitchellh/reflectwalk v1.0.2 // indirect
diff --git a/go.sum b/go.sum
index 35a95d12d15..bd298867ec7 100644
--- a/go.sum
+++ b/go.sum
@@ -68,8 +68,10 @@ github.com/aws/smithy-go v1.19.0 h1:KWFKQV80DpP3vJrrA9sVAHQ5gc2z8i4EzrLhLlWXcBM=
 github.com/aws/smithy-go v1.19.0/go.mod h1:NukqUGpCZIILqqiV0NIjeFh24kd/FAa4beRb6nbIUPE=
 github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM=
 github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw=
-github.com/caddyserver/certmagic v0.20.0 h1:bTw7LcEZAh9ucYCRXyCpIrSAGplplI0vGYJ4BpCQ/Fc=
-github.com/caddyserver/certmagic v0.20.0/go.mod h1:N4sXgpICQUskEWpj7zVzvWD41p3NYacrNoZYiRM2jTg=
+github.com/caddyserver/certmagic v0.20.1-0.20240412214119-167015dd6570 h1:SsAXjoQx2wOmLl6mEwJEwh7wwys2hb/l/mhtmxA3wts=
+github.com/caddyserver/certmagic v0.20.1-0.20240412214119-167015dd6570/go.mod h1:e1NhB1rF5KZnAuAX6oSyhE7sg1Ru5bWgggw5RtauhEY=
+github.com/caddyserver/zerossl v0.1.2 h1:tlEu1VzWGoqcCpivs9liKAKhfpJWYJkHEMmlxRbVAxE=
+github.com/caddyserver/zerossl v0.1.2/go.mod h1:wtiJEHbdvunr40ZzhXlnIkOB8Xj4eKtBKizCcZitJiQ=
 github.com/cenkalti/backoff/v4 v4.2.1 h1:y4OZtCnogmCPw98Zjyt5a6+QwPLGkiQsYW5oUqylYbM=
 github.com/cenkalti/backoff/v4 v4.2.1/go.mod h1:Y3VNntkOUPxTVeUxJ/G5vcM//AlwfmyYozVcomhLiZE=
 github.com/cespare/xxhash v1.1.0 h1:a6HrQnmkObjyL+Gs60czilIUGqrzKutQD6XZog3p+ko=
@@ -256,8 +258,8 @@ github.com/klauspost/compress v1.12.3/go.mod h1:8dP1Hq4DHOhN9w426knH3Rhby4rFm6D8
 github.com/klauspost/compress v1.17.0 h1:Rnbp4K9EjcDuVuHtd0dgA4qNuv9yKDYKK1ulpJwgrqM=
 github.com/klauspost/compress v1.17.0/go.mod h1:ntbaceVETuRiXiv4DpjP66DpAtAGkEQskQzEyD//IeE=
 github.com/klauspost/cpuid/v2 v2.0.12/go.mod h1:g2LTdtYhdyuGPqyWyv7qRAmj1WBqxuObKfj5c0PQa7c=
-github.com/klauspost/cpuid/v2 v2.2.5 h1:0E5MSMDEoAulmXNFquVs//DdoomxaoTY1kUhbc/qbZg=
-github.com/klauspost/cpuid/v2 v2.2.5/go.mod h1:Lcz8mBdAVJIBVzewtcLocK12l3Y+JytZYpaMropDUws=
+github.com/klauspost/cpuid/v2 v2.2.7 h1:ZWSB3igEs+d0qvnxR/ZBzXVmxkgt8DdzP6m9pfuVLDM=
+github.com/klauspost/cpuid/v2 v2.2.7/go.mod h1:Lcz8mBdAVJIBVzewtcLocK12l3Y+JytZYpaMropDUws=
 github.com/konsorten/go-windows-terminal-sequences v1.0.1/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ=
 github.com/konsorten/go-windows-terminal-sequences v1.0.2/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ=
 github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo=
@@ -275,8 +277,8 @@ github.com/lib/pq v1.2.0/go.mod h1:5WUZQaWbwv1U+lTReE5YruASi9Al49XbQIvNi/34Woo=
 github.com/lib/pq v1.10.2/go.mod h1:AlVN5x4E4T544tWzH6hKfbfQvm3HdbOxrmggDNAPY9o=
 github.com/lib/pq v1.10.9 h1:YXG7RB+JIjhP29X+OtkiDnYaXQwpS4JEWq7dtCCRUEw=
 github.com/lib/pq v1.10.9/go.mod h1:AlVN5x4E4T544tWzH6hKfbfQvm3HdbOxrmggDNAPY9o=
-github.com/libdns/libdns v0.2.1 h1:Wu59T7wSHRgtA0cfxC+n1c/e+O3upJGWytknkmFEDis=
-github.com/libdns/libdns v0.2.1/go.mod h1:yQCXzk1lEZmmCPa857bnk4TsOiqYasqpyOEeSObbb40=
+github.com/libdns/libdns v0.2.2 h1:O6ws7bAfRPaBsgAYt8MDe2HcNBGC29hkZ9MX2eUSX3s=
+github.com/libdns/libdns v0.2.2/go.mod h1:4Bj9+5CQiNMVGf87wjX4CY3HQJypUHRuLvlsfsZqLWQ=
 github.com/magiconair/properties v1.8.0/go.mod h1:PppfXfuXeibc/6YijjN8zIbojt8czPbwD3XqdrwzmxQ=
 github.com/manifoldco/promptui v0.9.0 h1:3V4HzJk1TtXW1MTZMP7mdlwbBpIinw3HztaIlYthEiA=
 github.com/manifoldco/promptui v0.9.0/go.mod h1:ka04sppxSGFAtxX0qhlYQjISsg9mR4GWtQEhdbn6Pgg=
@@ -292,10 +294,10 @@ github.com/mattn/go-isatty v0.0.20 h1:xfD0iDuEKnDkl03q4limB+vH+GxLEtL/jb4xVJSWWE
 github.com/mattn/go-isatty v0.0.20/go.mod h1:W+V8PltTTMOvKvAeJH7IuucS94S2C6jfK/D7dTCTo3Y=
 github.com/mgutz/ansi v0.0.0-20200706080929-d51e80ef957d h1:5PJl274Y63IEHC+7izoQE9x6ikvDFZS2mDVS3drnohI=
 github.com/mgutz/ansi v0.0.0-20200706080929-d51e80ef957d/go.mod h1:01TrycV0kFyexm33Z7vhZRXopbI8J3TDReVlkTgMUxE=
-github.com/mholt/acmez v1.2.0 h1:1hhLxSgY5FvH5HCnGUuwbKY2VQVo8IU7rxXKSnZ7F30=
-github.com/mholt/acmez v1.2.0/go.mod h1:VT9YwH1xgNX1kmYY89gY8xPJC84BFAisjo8Egigt4kE=
-github.com/miekg/dns v1.1.55 h1:GoQ4hpsj0nFLYe+bWiCToyrBEJXkQfOOIvFGFy0lEgo=
-github.com/miekg/dns v1.1.55/go.mod h1:uInx36IzPl7FYnDcMeVWxj9byh7DutNykX4G9Sj60FY=
+github.com/mholt/acmez/v2 v2.0.0-beta.2 h1:GIgGILx8AWN0ePyTd+bjs2WDgNiIWm0nBwDLWp59aHc=
+github.com/mholt/acmez/v2 v2.0.0-beta.2/go.mod h1:fX4c9r5jYwMyMsC+7tkYRxHibkOTgta5DIFGoe67e1U=
+github.com/miekg/dns v1.1.58 h1:ca2Hdkz+cDg/7eNF6V56jjzuZ4aCAE+DbVkILdQWG/4=
+github.com/miekg/dns v1.1.58/go.mod h1:Ypv+3b/KadlvW9vJfXOTf300O4UqaHFzFCuHz+rPkBY=
 github.com/mitchellh/copystructure v1.0.0/go.mod h1:SNtv71yrdKgLRyLFxmLdkAbkKEFWgYaq1OVrnRcwhnw=
 github.com/mitchellh/copystructure v1.2.0 h1:vpKXTN4ewci03Vljg/q9QvCGUDttBOGBIa15WveJJGw=
 github.com/mitchellh/copystructure v1.2.0/go.mod h1:qLl+cE2AmVv+CoeAwDPye/v+N2HKCj9FbZEVFJRxO9s=
diff --git a/modules/caddyhttp/autohttps.go b/modules/caddyhttp/autohttps.go
index e28062f0514..54a2d9ccd22 100644
--- a/modules/caddyhttp/autohttps.go
+++ b/modules/caddyhttp/autohttps.go
@@ -287,6 +287,16 @@ uniqueDomainsLoop:
 			for _, ap := range app.tlsApp.Automation.Policies {
 				for _, apHost := range ap.Subjects() {
 					if apHost == d {
+						// if the automation policy has all internal subjects but no issuers,
+						// it will default to CertMagic's issuers which are public CAs; use
+						// our internal issuer instead
+						if len(ap.Issuers) == 0 && ap.AllInternalSubjects() {
+							iss := new(caddytls.InternalIssuer)
+							if err := iss.Provision(ctx); err != nil {
+								return err
+							}
+							ap.Issuers = append(ap.Issuers, iss)
+						}
 						continue uniqueDomainsLoop
 					}
 				}
diff --git a/modules/caddytls/acmeissuer.go b/modules/caddytls/acmeissuer.go
index 8a7f8b49910..a14dc61a85e 100644
--- a/modules/caddytls/acmeissuer.go
+++ b/modules/caddytls/acmeissuer.go
@@ -17,14 +17,19 @@ package caddytls
 import (
 	"context"
 	"crypto/x509"
+	"encoding/json"
 	"fmt"
+	"net/http"
+	"net/url"
 	"os"
 	"strconv"
+	"strings"
 	"time"
 
 	"github.com/caddyserver/certmagic"
-	"github.com/mholt/acmez"
-	"github.com/mholt/acmez/acme"
+	"github.com/caddyserver/zerossl"
+	"github.com/mholt/acmez/v2"
+	"github.com/mholt/acmez/v2/acme"
 	"go.uber.org/zap"
 
 	"github.com/caddyserver/caddy/v2"
@@ -142,12 +147,14 @@ func (iss *ACMEIssuer) Provision(ctx caddy.Context) error {
 			iss.Challenges.DNS.solver = deprecatedProvider
 		} else {
 			iss.Challenges.DNS.solver = &certmagic.DNS01Solver{
-				DNSProvider:        val.(certmagic.ACMEDNSProvider),
-				TTL:                time.Duration(iss.Challenges.DNS.TTL),
-				PropagationDelay:   time.Duration(iss.Challenges.DNS.PropagationDelay),
-				PropagationTimeout: time.Duration(iss.Challenges.DNS.PropagationTimeout),
-				Resolvers:          iss.Challenges.DNS.Resolvers,
-				OverrideDomain:     iss.Challenges.DNS.OverrideDomain,
+				DNSManager: certmagic.DNSManager{
+					DNSProvider:        val.(certmagic.DNSProvider),
+					TTL:                time.Duration(iss.Challenges.DNS.TTL),
+					PropagationDelay:   time.Duration(iss.Challenges.DNS.PropagationDelay),
+					PropagationTimeout: time.Duration(iss.Challenges.DNS.PropagationTimeout),
+					Resolvers:          iss.Challenges.DNS.Resolvers,
+					OverrideDomain:     iss.Challenges.DNS.OverrideDomain,
+				},
 			}
 		}
 	}
@@ -210,6 +217,18 @@ func (iss *ACMEIssuer) makeIssuerTemplate() (certmagic.ACMEIssuer, error) {
 		}
 	}
 
+	// ZeroSSL requires EAB, but we can generate that automatically (requires an email address be configured)
+	if strings.HasPrefix(iss.CA, "https://acme.zerossl.com/") {
+		template.NewAccountFunc = func(ctx context.Context, acmeIss *certmagic.ACMEIssuer, acct acme.Account) (acme.Account, error) {
+			if acmeIss.ExternalAccount != nil {
+				return acct, nil
+			}
+			var err error
+			acmeIss.ExternalAccount, acct, err = iss.generateZeroSSLEABCredentials(ctx, acct)
+			return acct, err
+		}
+	}
+
 	return template, nil
 }
 
@@ -248,6 +267,65 @@ func (iss *ACMEIssuer) Revoke(ctx context.Context, cert certmagic.CertificateRes
 // to be accessed and manipulated.
 func (iss *ACMEIssuer) GetACMEIssuer() *ACMEIssuer { return iss }
 
+// generateZeroSSLEABCredentials generates ZeroSSL EAB credentials for the primary contact email
+// on the issuer. It should only be usedif the CA endpoint is ZeroSSL. An email address is required.
+func (iss *ACMEIssuer) generateZeroSSLEABCredentials(ctx context.Context, acct acme.Account) (*acme.EAB, acme.Account, error) {
+	if strings.TrimSpace(iss.Email) == "" {
+		return nil, acme.Account{}, fmt.Errorf("your email address is required to use ZeroSSL's ACME endpoint")
+	}
+
+	if len(acct.Contact) == 0 {
+		// we borrow the email from config or the default email, so ensure it's saved with the account
+		acct.Contact = []string{"mailto:" + iss.Email}
+	}
+
+	endpoint := zerossl.BaseURL + "/acme/eab-credentials-email"
+	form := url.Values{"email": []string{iss.Email}}
+	body := strings.NewReader(form.Encode())
+
+	req, err := http.NewRequestWithContext(ctx, http.MethodPost, endpoint, body)
+	if err != nil {
+		return nil, acct, fmt.Errorf("forming request: %v", err)
+	}
+	req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
+	req.Header.Set("User-Agent", certmagic.UserAgent)
+
+	resp, err := http.DefaultClient.Do(req)
+	if err != nil {
+		return nil, acct, fmt.Errorf("performing EAB credentials request: %v", err)
+	}
+	defer resp.Body.Close()
+
+	var result struct {
+		Success bool `json:"success"`
+		Error   struct {
+			Code int    `json:"code"`
+			Type string `json:"type"`
+		} `json:"error"`
+		EABKID     string `json:"eab_kid"`
+		EABHMACKey string `json:"eab_hmac_key"`
+	}
+	err = json.NewDecoder(resp.Body).Decode(&result)
+	if err != nil {
+		return nil, acct, fmt.Errorf("decoding API response: %v", err)
+	}
+	if result.Error.Code != 0 {
+		// do this check first because ZeroSSL's API returns 200 on errors
+		return nil, acct, fmt.Errorf("failed getting EAB credentials: HTTP %d: %s (code %d)",
+			resp.StatusCode, result.Error.Type, result.Error.Code)
+	}
+	if resp.StatusCode != http.StatusOK {
+		return nil, acct, fmt.Errorf("failed getting EAB credentials: HTTP %d", resp.StatusCode)
+	}
+
+	iss.logger.Info("generated EAB credentials", zap.String("key_id", result.EABKID))
+
+	return &acme.EAB{
+		KeyID:  result.EABKID,
+		MACKey: result.EABHMACKey,
+	}, acct, nil
+}
+
 // UnmarshalCaddyfile deserializes Caddyfile tokens into iss.
 //
 //	... acme [<directory_url>] {
diff --git a/modules/caddytls/automation.go b/modules/caddytls/automation.go
index a90e5ded803..803cb174e38 100644
--- a/modules/caddytls/automation.go
+++ b/modules/caddytls/automation.go
@@ -24,7 +24,7 @@ import (
 	"strings"
 
 	"github.com/caddyserver/certmagic"
-	"github.com/mholt/acmez"
+	"github.com/mholt/acmez/v2"
 	"go.uber.org/zap"
 
 	"github.com/caddyserver/caddy/v2"
@@ -201,6 +201,7 @@ func (ap *AutomationPolicy) Provision(tlsApp *TLS) error {
 	// store them on the policy before putting it on the config
 
 	// load and provision any cert manager modules
+	hadExplicitManagers := len(ap.ManagersRaw) > 0
 	if ap.ManagersRaw != nil {
 		vals, err := tlsApp.ctx.LoadModule(ap, "ManagersRaw")
 		if err != nil {
@@ -256,12 +257,25 @@ func (ap *AutomationPolicy) Provision(tlsApp *TLS) error {
 	if ap.OnDemand || len(ap.Managers) > 0 {
 		// permission module is now required after a number of negligence cases that allowed abuse;
 		// but it may still be optional for explicit subjects (bounded, non-wildcard), for the
-		// internal issuer since it doesn't cause public PKI pressure on ACME servers
-		if ap.isWildcardOrDefault() && !ap.onlyInternalIssuer() && (tlsApp.Automation == nil || tlsApp.Automation.OnDemand == nil || tlsApp.Automation.OnDemand.permission == nil) {
-			return fmt.Errorf("on-demand TLS cannot be enabled without a permission module to prevent abuse; please refer to documentation for details")
+		// internal issuer since it doesn't cause public PKI pressure on ACME servers; subtly, it
+		// is useful to allow on-demand TLS to be enabled so Managers can be used, but to still
+		// prevent issuance from Issuers (when Managers don't provide a certificate) if there's no
+		// permission module configured
+		noProtections := ap.isWildcardOrDefault() && !ap.onlyInternalIssuer() && (tlsApp.Automation == nil || tlsApp.Automation.OnDemand == nil || tlsApp.Automation.OnDemand.permission == nil)
+		failClosed := noProtections && hadExplicitManagers // don't allow on-demand issuance (other than implicit managers) if no managers have been explicitly configured
+		if noProtections {
+			if !hadExplicitManagers {
+				// no managers, no explicitly-configured permission module, this is a config error
+				return fmt.Errorf("on-demand TLS cannot be enabled without a permission module to prevent abuse; please refer to documentation for details")
+			}
+			// allow on-demand to be enabled but only for the purpose of the Managers; issuance won't be allowed from Issuers
+			tlsApp.logger.Warn("on-demand TLS can only get certificates from the configured external manager(s) because no ask endpoint / permission module is specified")
 		}
 		ond = &certmagic.OnDemandConfig{
 			DecisionFunc: func(ctx context.Context, name string) error {
+				if failClosed {
+					return fmt.Errorf("no permission module configured; certificates not allowed except from external Managers")
+				}
 				if tlsApp.Automation == nil || tlsApp.Automation.OnDemand == nil {
 					return nil
 				}
@@ -344,6 +358,16 @@ func (ap *AutomationPolicy) Subjects() []string {
 	return ap.subjects
 }
 
+// AllInternalSubjects returns true if all the subjects on this policy are internal.
+func (ap *AutomationPolicy) AllInternalSubjects() bool {
+	for _, subj := range ap.subjects {
+		if !certmagic.SubjectIsInternal(subj) {
+			return false
+		}
+	}
+	return true
+}
+
 func (ap *AutomationPolicy) onlyInternalIssuer() bool {
 	if len(ap.Issuers) != 1 {
 		return false
@@ -370,17 +394,21 @@ func (ap *AutomationPolicy) isWildcardOrDefault() bool {
 
 // DefaultIssuers returns empty Issuers (not provisioned) to be used as defaults.
 // This function is experimental and has no compatibility promises.
-func DefaultIssuers() []certmagic.Issuer {
-	return []certmagic.Issuer{
-		new(ACMEIssuer),
-		&ZeroSSLIssuer{ACMEIssuer: new(ACMEIssuer)},
+func DefaultIssuers(userEmail string) []certmagic.Issuer {
+	issuers := []certmagic.Issuer{new(ACMEIssuer)}
+	if strings.TrimSpace(userEmail) != "" {
+		issuers = append(issuers, &ACMEIssuer{
+			CA:    certmagic.ZeroSSLProductionCA,
+			Email: userEmail,
+		})
 	}
+	return issuers
 }
 
 // DefaultIssuersProvisioned returns empty but provisioned default Issuers from
 // DefaultIssuers(). This function is experimental and has no compatibility promises.
 func DefaultIssuersProvisioned(ctx caddy.Context) ([]certmagic.Issuer, error) {
-	issuers := DefaultIssuers()
+	issuers := DefaultIssuers("")
 	for i, iss := range issuers {
 		if prov, ok := iss.(caddy.Provisioner); ok {
 			err := prov.Provision(ctx)
@@ -453,6 +481,7 @@ type TLSALPNChallengeConfig struct {
 type DNSChallengeConfig struct {
 	// The DNS provider module to use which will manage
 	// the DNS records relevant to the ACME challenge.
+	// Required.
 	ProviderRaw json.RawMessage `json:"provider,omitempty" caddy:"namespace=dns.providers inline_key=name"`
 
 	// The TTL of the TXT record used for the DNS challenge.
diff --git a/modules/caddytls/capools.go b/modules/caddytls/capools.go
index 44a2fa2c2f9..7e378aac45c 100644
--- a/modules/caddytls/capools.go
+++ b/modules/caddytls/capools.go
@@ -579,8 +579,7 @@ func (hcp *HTTPCertPool) Provision(ctx caddy.Context) error {
 		customTransport.TLSClientConfig = tlsConfig
 	}
 
-	var httpClient *http.Client
-	*httpClient = *http.DefaultClient
+	httpClient := *http.DefaultClient
 	httpClient.Transport = customTransport
 
 	for _, uri := range hcp.Endpoints {
diff --git a/modules/caddytls/certmanagers.go b/modules/caddytls/certmanagers.go
index 9bb436a3746..b2e2eb07342 100644
--- a/modules/caddytls/certmanagers.go
+++ b/modules/caddytls/certmanagers.go
@@ -96,6 +96,11 @@ type HTTPCertGetter struct {
 	// To be valid, the response must be HTTP 200 with a PEM body
 	// consisting of blocks for the certificate chain and the private
 	// key.
+	//
+	// To indicate that this manager is not managing a certificate for
+	// the described handshake, the endpoint should return HTTP 204
+	// (No Content). Error statuses will indicate that the manager is
+	// capable of providing a certificate but was unable to.
 	URL string `json:"url,omitempty"`
 
 	ctx context.Context
@@ -147,6 +152,10 @@ func (hcg HTTPCertGetter) GetCertificate(ctx context.Context, hello *tls.ClientH
 		return nil, err
 	}
 	defer resp.Body.Close()
+	if resp.StatusCode == http.StatusNoContent {
+		// endpoint is not managing certs for this handshake
+		return nil, nil
+	}
 	if resp.StatusCode != http.StatusOK {
 		return nil, fmt.Errorf("got HTTP %d", resp.StatusCode)
 	}
diff --git a/modules/caddytls/connpolicy.go b/modules/caddytls/connpolicy.go
index 49c7add49d3..1e145544602 100644
--- a/modules/caddytls/connpolicy.go
+++ b/modules/caddytls/connpolicy.go
@@ -26,7 +26,7 @@ import (
 	"path/filepath"
 	"strings"
 
-	"github.com/mholt/acmez"
+	"github.com/mholt/acmez/v2"
 	"go.uber.org/zap"
 
 	"github.com/caddyserver/caddy/v2"
diff --git a/modules/caddytls/tls.go b/modules/caddytls/tls.go
index bb4cd971537..2a05d5235f7 100644
--- a/modules/caddytls/tls.go
+++ b/modules/caddytls/tls.go
@@ -176,8 +176,9 @@ func (t *TLS) Provision(ctx caddy.Context) error {
 		t.Automation.OnDemand.permission = val.(OnDemandPermission)
 	}
 
-	// on-demand rate limiting
+	// on-demand rate limiting (TODO: deprecated, and should be removed later; rate limiting is ineffective now that permission modules are required)
 	if t.Automation != nil && t.Automation.OnDemand != nil && t.Automation.OnDemand.RateLimit != nil {
+		t.logger.Warn("DEPRECATED: on_demand.rate_limit will be removed in a future release; use permission modules or external certificate managers instead")
 		onDemandRateLimiter.SetMaxEvents(t.Automation.OnDemand.RateLimit.Burst)
 		onDemandRateLimiter.SetWindow(time.Duration(t.Automation.OnDemand.RateLimit.Interval))
 	} else {
@@ -413,35 +414,49 @@ func (t *TLS) Manage(names []string) error {
 	return nil
 }
 
-// HandleHTTPChallenge ensures that the HTTP challenge is handled for the
-// certificate named by r.Host, if it is an HTTP challenge request. It
-// requires that the automation policy for r.Host has an issuer of type
-// *certmagic.ACMEManager, or one that is ACME-enabled (GetACMEIssuer()).
+// HandleHTTPChallenge ensures that the ACME HTTP challenge or ZeroSSL HTTP
+// validation request is handled for the certificate named by r.Host, if it
+// is an HTTP challenge request. It requires that the automation policy for
+// r.Host has an issuer that implements GetACMEIssuer() or is a *ZeroSSLIssuer.
 func (t *TLS) HandleHTTPChallenge(w http.ResponseWriter, r *http.Request) bool {
+	acmeChallenge := certmagic.LooksLikeHTTPChallenge(r)
+	zerosslValidation := certmagic.LooksLikeZeroSSLHTTPValidation(r)
+
 	// no-op if it's not an ACME challenge request
-	if !certmagic.LooksLikeHTTPChallenge(r) {
+	if !acmeChallenge && !zerosslValidation {
 		return false
 	}
 
 	// try all the issuers until we find the one that initiated the challenge
 	ap := t.getAutomationPolicyForName(r.Host)
-	type acmeCapable interface{ GetACMEIssuer() *ACMEIssuer }
-	for _, iss := range ap.magic.Issuers {
-		if am, ok := iss.(acmeCapable); ok {
-			iss := am.GetACMEIssuer()
-			if iss.issuer.HandleHTTPChallenge(w, r) {
-				return true
+
+	if acmeChallenge {
+		type acmeCapable interface{ GetACMEIssuer() *ACMEIssuer }
+
+		for _, iss := range ap.magic.Issuers {
+			if acmeIssuer, ok := iss.(acmeCapable); ok {
+				if acmeIssuer.GetACMEIssuer().issuer.HandleHTTPChallenge(w, r) {
+					return true
+				}
 			}
 		}
-	}
 
-	// it's possible another server in this process initiated the challenge;
-	// users have requested that Caddy only handle HTTP challenges it initiated,
-	// so that users can proxy the others through to their backends; but we
-	// might not have an automation policy for all identifiers that are trying
-	// to get certificates (e.g. the admin endpoint), so we do this manual check
-	if challenge, ok := certmagic.GetACMEChallenge(r.Host); ok {
-		return certmagic.SolveHTTPChallenge(t.logger, w, r, challenge.Challenge)
+		// it's possible another server in this process initiated the challenge;
+		// users have requested that Caddy only handle HTTP challenges it initiated,
+		// so that users can proxy the others through to their backends; but we
+		// might not have an automation policy for all identifiers that are trying
+		// to get certificates (e.g. the admin endpoint), so we do this manual check
+		if challenge, ok := certmagic.GetACMEChallenge(r.Host); ok {
+			return certmagic.SolveHTTPChallenge(t.logger, w, r, challenge.Challenge)
+		}
+	} else if zerosslValidation {
+		for _, iss := range ap.magic.Issuers {
+			if ziss, ok := iss.(*ZeroSSLIssuer); ok {
+				if ziss.issuer.HandleZeroSSLHTTPValidation(w, r) {
+					return true
+				}
+			}
+		}
 	}
 
 	return false
diff --git a/modules/caddytls/zerosslissuer.go b/modules/caddytls/zerosslissuer.go
index 1c091a076e8..b8727ab669d 100644
--- a/modules/caddytls/zerosslissuer.go
+++ b/modules/caddytls/zerosslissuer.go
@@ -17,19 +17,15 @@ package caddytls
 import (
 	"context"
 	"crypto/x509"
-	"encoding/json"
 	"fmt"
-	"io"
-	"net/http"
-	"net/url"
-	"strings"
-	"sync"
+	"strconv"
+	"time"
 
 	"github.com/caddyserver/certmagic"
-	"github.com/mholt/acmez/acme"
 	"go.uber.org/zap"
 
 	"github.com/caddyserver/caddy/v2"
+	"github.com/caddyserver/caddy/v2/caddyconfig"
 	"github.com/caddyserver/caddy/v2/caddyconfig/caddyfile"
 )
 
@@ -37,24 +33,36 @@ func init() {
 	caddy.RegisterModule(new(ZeroSSLIssuer))
 }
 
-// ZeroSSLIssuer makes an ACME issuer for getting certificates
-// from ZeroSSL by automatically generating EAB credentials.
-// Please be sure to set a valid email address in your config
-// so you can access/manage your domains in your ZeroSSL account.
-//
-// This issuer is only needed for automatic generation of EAB
-// credentials. If manually configuring/reusing EAB credentials,
-// the standard ACMEIssuer may be used if desired.
+// ZeroSSLIssuer uses the ZeroSSL API to get certificates.
+// Note that this is distinct from ZeroSSL's ACME endpoint.
+// To use ZeroSSL's ACME endpoint, use the ACMEIssuer
+// configured with ZeroSSL's ACME directory endpoint.
 type ZeroSSLIssuer struct {
-	*ACMEIssuer
-
 	// The API key (or "access key") for using the ZeroSSL API.
-	// This is optional, but can be used if you have an API key
-	// already and don't want to supply your email address.
+	// REQUIRED.
 	APIKey string `json:"api_key,omitempty"`
 
-	mu     sync.Mutex
-	logger *zap.Logger
+	// How many days the certificate should be valid for.
+	// Only certain values are accepted; see ZeroSSL docs.
+	ValidityDays int `json:"validity_days,omitempty"`
+
+	// The host to bind to when opening a listener for
+	// verifying domain names (or IPs).
+	ListenHost string `json:"listen_host,omitempty"`
+
+	// If HTTP is forwarded from port 80, specify the
+	// forwarded port here.
+	AlternateHTTPPort int `json:"alternate_http_port,omitempty"`
+
+	// Use CNAME validation instead of HTTP. ZeroSSL's
+	// API uses CNAME records for DNS validation, similar
+	// to how Let's Encrypt uses TXT records for the
+	// DNS challenge.
+	CNAMEValidation *DNSChallengeConfig `json:"cname_validation,omitempty"`
+
+	logger  *zap.Logger
+	storage certmagic.Storage
+	issuer  *certmagic.ZeroSSLIssuer
 }
 
 // CaddyModule returns the Caddy module information.
@@ -65,178 +73,184 @@ func (*ZeroSSLIssuer) CaddyModule() caddy.ModuleInfo {
 	}
 }
 
-// Provision sets up iss.
+// Provision sets up the issuer.
 func (iss *ZeroSSLIssuer) Provision(ctx caddy.Context) error {
 	iss.logger = ctx.Logger()
-	if iss.ACMEIssuer == nil {
-		iss.ACMEIssuer = new(ACMEIssuer)
-	}
-	if iss.ACMEIssuer.CA == "" {
-		iss.ACMEIssuer.CA = certmagic.ZeroSSLProductionCA
-	}
-	return iss.ACMEIssuer.Provision(ctx)
-}
+	iss.storage = ctx.Storage()
+	repl := caddy.NewReplacer()
 
-// newAccountCallback generates EAB if not already provided. It also sets a valid default contact on the account if not set.
-func (iss *ZeroSSLIssuer) newAccountCallback(ctx context.Context, acmeIss *certmagic.ACMEIssuer, acct acme.Account) (acme.Account, error) {
-	if acmeIss.ExternalAccount != nil {
-		return acct, nil
-	}
-	var err error
-	acmeIss.ExternalAccount, acct, err = iss.generateEABCredentials(ctx, acct)
-	return acct, err
-}
-
-// generateEABCredentials generates EAB credentials using the API key if provided,
-// otherwise using the primary contact email on the issuer. If an email is not set
-// on the issuer, a default generic email is used.
-func (iss *ZeroSSLIssuer) generateEABCredentials(ctx context.Context, acct acme.Account) (*acme.EAB, acme.Account, error) {
-	var endpoint string
-	var body io.Reader
-
-	// there are two ways to generate EAB credentials: authenticated with
-	// their API key, or unauthenticated with their email address
-	if iss.APIKey != "" {
-		apiKey := caddy.NewReplacer().ReplaceAll(iss.APIKey, "")
-		if apiKey == "" {
-			return nil, acct, fmt.Errorf("missing API key: '%v'", iss.APIKey)
-		}
-		qs := url.Values{"access_key": []string{apiKey}}
-		endpoint = fmt.Sprintf("%s/eab-credentials?%s", zerosslAPIBase, qs.Encode())
-	} else {
-		email := iss.Email
-		if email == "" {
-			iss.logger.Warn("missing email address for ZeroSSL; it is strongly recommended to set one for next time")
-			email = "caddy@zerossl.com" // special email address that preserves backwards-compat, but which black-holes dashboard features, oh well
+	var dnsManager *certmagic.DNSManager
+	if iss.CNAMEValidation != nil && len(iss.CNAMEValidation.ProviderRaw) > 0 {
+		val, err := ctx.LoadModule(iss.CNAMEValidation, "ProviderRaw")
+		if err != nil {
+			return fmt.Errorf("loading DNS provider module: %v", err)
 		}
-		if len(acct.Contact) == 0 {
-			// we borrow the email from config or the default email, so ensure it's saved with the account
-			acct.Contact = []string{"mailto:" + email}
+		dnsManager = &certmagic.DNSManager{
+			DNSProvider:        val.(certmagic.DNSProvider),
+			TTL:                time.Duration(iss.CNAMEValidation.TTL),
+			PropagationDelay:   time.Duration(iss.CNAMEValidation.PropagationDelay),
+			PropagationTimeout: time.Duration(iss.CNAMEValidation.PropagationTimeout),
+			Resolvers:          iss.CNAMEValidation.Resolvers,
+			OverrideDomain:     iss.CNAMEValidation.OverrideDomain,
+			Logger:             iss.logger.Named("cname"),
 		}
-		endpoint = zerosslAPIBase + "/eab-credentials-email"
-		form := url.Values{"email": []string{email}}
-		body = strings.NewReader(form.Encode())
-	}
-
-	req, err := http.NewRequestWithContext(ctx, http.MethodPost, endpoint, body)
-	if err != nil {
-		return nil, acct, fmt.Errorf("forming request: %v", err)
-	}
-	if body != nil {
-		req.Header.Set("Content-Type", "application/x-www-form-urlencoded")
-	}
-	req.Header.Set("User-Agent", certmagic.UserAgent)
-
-	resp, err := http.DefaultClient.Do(req)
-	if err != nil {
-		return nil, acct, fmt.Errorf("performing EAB credentials request: %v", err)
-	}
-	defer resp.Body.Close()
-
-	var result struct {
-		Success bool `json:"success"`
-		Error   struct {
-			Code int    `json:"code"`
-			Type string `json:"type"`
-		} `json:"error"`
-		EABKID     string `json:"eab_kid"`
-		EABHMACKey string `json:"eab_hmac_key"`
-	}
-	err = json.NewDecoder(resp.Body).Decode(&result)
-	if err != nil {
-		return nil, acct, fmt.Errorf("decoding API response: %v", err)
-	}
-	if result.Error.Code != 0 {
-		return nil, acct, fmt.Errorf("failed getting EAB credentials: HTTP %d: %s (code %d)",
-			resp.StatusCode, result.Error.Type, result.Error.Code)
 	}
-	if resp.StatusCode != http.StatusOK {
-		return nil, acct, fmt.Errorf("failed getting EAB credentials: HTTP %d", resp.StatusCode)
-	}
-
-	iss.logger.Info("generated EAB credentials", zap.String("key_id", result.EABKID))
-
-	return &acme.EAB{
-		KeyID:  result.EABKID,
-		MACKey: result.EABHMACKey,
-	}, acct, nil
-}
 
-// initialize modifies the template for the underlying ACMEIssuer
-// values by setting the CA endpoint to the ZeroSSL directory and
-// setting the NewAccountFunc callback to one which allows us to
-// generate EAB credentials only if a new account is being made.
-// Since it modifies the stored template, its effect should only
-// be needed once, but it is fine to call it repeatedly.
-func (iss *ZeroSSLIssuer) initialize() {
-	iss.mu.Lock()
-	defer iss.mu.Unlock()
-	if iss.ACMEIssuer.issuer.NewAccountFunc == nil {
-		iss.ACMEIssuer.issuer.NewAccountFunc = iss.newAccountCallback
+	iss.issuer = &certmagic.ZeroSSLIssuer{
+		APIKey:          repl.ReplaceAll(iss.APIKey, ""),
+		ValidityDays:    iss.ValidityDays,
+		ListenHost:      iss.ListenHost,
+		AltHTTPPort:     iss.AlternateHTTPPort,
+		Storage:         iss.storage,
+		CNAMEValidation: dnsManager,
+		Logger:          iss.logger,
 	}
-}
 
-// PreCheck implements the certmagic.PreChecker interface.
-func (iss *ZeroSSLIssuer) PreCheck(ctx context.Context, names []string, interactive bool) error {
-	iss.initialize()
-	return iss.ACMEIssuer.PreCheck(ctx, names, interactive)
+	return nil
 }
 
 // Issue obtains a certificate for the given csr.
 func (iss *ZeroSSLIssuer) Issue(ctx context.Context, csr *x509.CertificateRequest) (*certmagic.IssuedCertificate, error) {
-	iss.initialize()
-	return iss.ACMEIssuer.Issue(ctx, csr)
+	return iss.issuer.Issue(ctx, csr)
 }
 
 // IssuerKey returns the unique issuer key for the configured CA endpoint.
 func (iss *ZeroSSLIssuer) IssuerKey() string {
-	iss.initialize()
-	return iss.ACMEIssuer.IssuerKey()
+	return iss.issuer.IssuerKey()
 }
 
 // Revoke revokes the given certificate.
 func (iss *ZeroSSLIssuer) Revoke(ctx context.Context, cert certmagic.CertificateResource, reason int) error {
-	iss.initialize()
-	return iss.ACMEIssuer.Revoke(ctx, cert, reason)
+	return iss.issuer.Revoke(ctx, cert, reason)
 }
 
 // UnmarshalCaddyfile deserializes Caddyfile tokens into iss.
 //
-//	... zerossl [<api_key>] {
-//	    ...
+//	... zerossl <api_key> {
+//		    validity_days <days>
+//		    alt_http_port <port>
+//		    dns <provider_name> ...
+//		    propagation_delay <duration>
+//		    propagation_timeout <duration>
+//		    resolvers <list...>
+//		    dns_ttl <duration>
 //	}
-//
-// Any of the subdirectives for the ACME issuer can be used in the block.
 func (iss *ZeroSSLIssuer) UnmarshalCaddyfile(d *caddyfile.Dispenser) error {
 	d.Next() // consume issuer name
+
+	// API key is required
+	if !d.NextArg() {
+		return d.ArgErr()
+	}
+	iss.APIKey = d.Val()
 	if d.NextArg() {
-		iss.APIKey = d.Val()
-		if d.NextArg() {
-			return d.ArgErr()
-		}
+		return d.ArgErr()
 	}
 
-	if iss.ACMEIssuer == nil {
-		iss.ACMEIssuer = new(ACMEIssuer)
-	}
-	err := iss.ACMEIssuer.UnmarshalCaddyfile(d.NewFromNextSegment())
-	if err != nil {
-		return err
+	for nesting := d.Nesting(); d.NextBlock(nesting); {
+		switch d.Val() {
+		case "validity_days":
+			if iss.ValidityDays != 0 {
+				return d.Errf("validity days is already specified: %d", iss.ValidityDays)
+			}
+			days, err := strconv.Atoi(d.Val())
+			if err != nil {
+				return d.Errf("invalid number of days %s: %v", d.Val(), err)
+			}
+			iss.ValidityDays = days
+
+		case "alt_http_port":
+			if !d.NextArg() {
+				return d.ArgErr()
+			}
+			port, err := strconv.Atoi(d.Val())
+			if err != nil {
+				return d.Errf("invalid port %s: %v", d.Val(), err)
+			}
+			iss.AlternateHTTPPort = port
+
+		case "dns":
+			if !d.NextArg() {
+				return d.ArgErr()
+			}
+			provName := d.Val()
+			if iss.CNAMEValidation == nil {
+				iss.CNAMEValidation = new(DNSChallengeConfig)
+			}
+			unm, err := caddyfile.UnmarshalModule(d, "dns.providers."+provName)
+			if err != nil {
+				return err
+			}
+			iss.CNAMEValidation.ProviderRaw = caddyconfig.JSONModuleObject(unm, "name", provName, nil)
+
+		case "propagation_delay":
+			if !d.NextArg() {
+				return d.ArgErr()
+			}
+			delayStr := d.Val()
+			delay, err := caddy.ParseDuration(delayStr)
+			if err != nil {
+				return d.Errf("invalid propagation_delay duration %s: %v", delayStr, err)
+			}
+			if iss.CNAMEValidation == nil {
+				iss.CNAMEValidation = new(DNSChallengeConfig)
+			}
+			iss.CNAMEValidation.PropagationDelay = caddy.Duration(delay)
+
+		case "propagation_timeout":
+			if !d.NextArg() {
+				return d.ArgErr()
+			}
+			timeoutStr := d.Val()
+			var timeout time.Duration
+			if timeoutStr == "-1" {
+				timeout = time.Duration(-1)
+			} else {
+				var err error
+				timeout, err = caddy.ParseDuration(timeoutStr)
+				if err != nil {
+					return d.Errf("invalid propagation_timeout duration %s: %v", timeoutStr, err)
+				}
+			}
+			if iss.CNAMEValidation == nil {
+				iss.CNAMEValidation = new(DNSChallengeConfig)
+			}
+			iss.CNAMEValidation.PropagationTimeout = caddy.Duration(timeout)
+
+		case "resolvers":
+			if iss.CNAMEValidation == nil {
+				iss.CNAMEValidation = new(DNSChallengeConfig)
+			}
+			iss.CNAMEValidation.Resolvers = d.RemainingArgs()
+			if len(iss.CNAMEValidation.Resolvers) == 0 {
+				return d.ArgErr()
+			}
+
+		case "dns_ttl":
+			if !d.NextArg() {
+				return d.ArgErr()
+			}
+			ttlStr := d.Val()
+			ttl, err := caddy.ParseDuration(ttlStr)
+			if err != nil {
+				return d.Errf("invalid dns_ttl duration %s: %v", ttlStr, err)
+			}
+			if iss.CNAMEValidation == nil {
+				iss.CNAMEValidation = new(DNSChallengeConfig)
+			}
+			iss.CNAMEValidation.TTL = caddy.Duration(ttl)
+
+		default:
+			return d.Errf("unrecognized zerossl issuer property: %s", d.Val())
+		}
 	}
+
 	return nil
 }
 
-const zerosslAPIBase = "https://api.zerossl.com/acme"
-
 // Interface guards
 var (
-	_ certmagic.PreChecker = (*ZeroSSLIssuer)(nil)
-	_ certmagic.Issuer     = (*ZeroSSLIssuer)(nil)
-	_ certmagic.Revoker    = (*ZeroSSLIssuer)(nil)
-	_ caddy.Provisioner    = (*ZeroSSLIssuer)(nil)
-	_ ConfigSetter         = (*ZeroSSLIssuer)(nil)
-
-	// a type which properly embeds an ACMEIssuer should implement
-	// this interface so it can be treated as an ACMEIssuer
-	_ interface{ GetACMEIssuer() *ACMEIssuer } = (*ZeroSSLIssuer)(nil)
+	_ certmagic.Issuer  = (*ZeroSSLIssuer)(nil)
+	_ certmagic.Revoker = (*ZeroSSLIssuer)(nil)
+	_ caddy.Provisioner = (*ZeroSSLIssuer)(nil)
 )


Test patch:
diff --git a/caddytest/integration/acme_test.go b/caddytest/integration/acme_test.go
index 840af023f67..ceacd1db0fa 100644
--- a/caddytest/integration/acme_test.go
+++ b/caddytest/integration/acme_test.go
@@ -13,8 +13,8 @@ import (
 
 	"github.com/caddyserver/caddy/v2"
 	"github.com/caddyserver/caddy/v2/caddytest"
-	"github.com/mholt/acmez"
-	"github.com/mholt/acmez/acme"
+	"github.com/mholt/acmez/v2"
+	"github.com/mholt/acmez/v2/acme"
 	smallstepacme "github.com/smallstep/certificates/acme"
 	"go.uber.org/zap"
 )
@@ -77,7 +77,7 @@ func TestACMEServerWithDefaults(t *testing.T) {
 		return
 	}
 
-	certs, err := client.ObtainCertificate(ctx, account, certPrivateKey, []string{"localhost"})
+	certs, err := client.ObtainCertificateForSANs(ctx, account, certPrivateKey, []string{"localhost"})
 	if err != nil {
 		t.Errorf("obtaining certificate: %v", err)
 		return
@@ -146,7 +146,7 @@ func TestACMEServerWithMismatchedChallenges(t *testing.T) {
 		return
 	}
 
-	certs, err := client.ObtainCertificate(ctx, account, certPrivateKey, []string{"localhost"})
+	certs, err := client.ObtainCertificateForSANs(ctx, account, certPrivateKey, []string{"localhost"})
 	if len(certs) > 0 {
 		t.Errorf("expected '0' certificates, but received '%d'", len(certs))
 	}
diff --git a/caddytest/integration/acmeserver_test.go b/caddytest/integration/acmeserver_test.go
index 435bfc7b48e..22b716f84db 100644
--- a/caddytest/integration/acmeserver_test.go
+++ b/caddytest/integration/acmeserver_test.go
@@ -9,8 +9,8 @@ import (
 	"testing"
 
 	"github.com/caddyserver/caddy/v2/caddytest"
-	"github.com/mholt/acmez"
-	"github.com/mholt/acmez/acme"
+	"github.com/mholt/acmez/v2"
+	"github.com/mholt/acmez/v2/acme"
 	"go.uber.org/zap"
 )
 
@@ -105,12 +105,7 @@ func TestACMEServerAllowPolicy(t *testing.T) {
 		return
 	}
 	{
-		certs, err := client.ObtainCertificate(
-			ctx,
-			account,
-			certPrivateKey,
-			[]string{"localhost"},
-		)
+		certs, err := client.ObtainCertificateForSANs(ctx, account, certPrivateKey, []string{"localhost"})
 		if err != nil {
 			t.Errorf("obtaining certificate for allowed domain: %v", err)
 			return
@@ -126,7 +121,7 @@ func TestACMEServerAllowPolicy(t *testing.T) {
 		}
 	}
 	{
-		_, err := client.ObtainCertificate(ctx, account, certPrivateKey, []string{"not-matching.localhost"})
+		_, err := client.ObtainCertificateForSANs(ctx, account, certPrivateKey, []string{"not-matching.localhost"})
 		if err == nil {
 			t.Errorf("obtaining certificate for 'not-matching.localhost' domain")
 		} else if err != nil && !strings.Contains(err.Error(), "urn:ietf:params:acme:error:rejectedIdentifier") {
@@ -199,7 +194,7 @@ func TestACMEServerDenyPolicy(t *testing.T) {
 		return
 	}
 	{
-		_, err := client.ObtainCertificate(ctx, account, certPrivateKey, []string{"deny.localhost"})
+		_, err := client.ObtainCertificateForSANs(ctx, account, certPrivateKey, []string{"deny.localhost"})
 		if err == nil {
 			t.Errorf("obtaining certificate for 'deny.localhost' domain")
 		} else if err != nil && !strings.Contains(err.Error(), "urn:ietf:params:acme:error:rejectedIdentifier") {
diff --git a/caddytest/integration/caddyfile_adapt/global_options_preferred_chains.caddyfiletest b/caddytest/integration/caddyfile_adapt/global_options_preferred_chains.caddyfiletest
index 9173b26bf40..1f5d0093e26 100644
--- a/caddytest/integration/caddyfile_adapt/global_options_preferred_chains.caddyfiletest
+++ b/caddytest/integration/caddyfile_adapt/global_options_preferred_chains.caddyfiletest
@@ -40,12 +40,6 @@ example.com
 								"preferred_chains": {
 									"smallest": true
 								}
-							},
-							{
-								"module": "zerossl",
-								"preferred_chains": {
-									"smallest": true
-								}
 							}
 						]
 					}
diff --git a/caddytest/integration/caddyfile_adapt/tls_automation_policies_3.caddyfiletest b/caddytest/integration/caddyfile_adapt/tls_automation_policies_3.caddyfiletest
index da5824a36b6..9daaf436d57 100644
--- a/caddytest/integration/caddyfile_adapt/tls_automation_policies_3.caddyfiletest
+++ b/caddytest/integration/caddyfile_adapt/tls_automation_policies_3.caddyfiletest
@@ -70,8 +70,9 @@ c.example.com {
 								"module": "acme"
 							},
 							{
+								"ca": "https://acme.zerossl.com/v2/DV90",
 								"email": "abc@example.com",
-								"module": "zerossl"
+								"module": "acme"
 							}
 						]
 					},
diff --git a/caddytest/integration/caddyfile_adapt/tls_automation_policies_4.caddyfiletest b/caddytest/integration/caddyfile_adapt/tls_automation_policies_4.caddyfiletest
index d8f2164de4a..a4385a8f3d7 100644
--- a/caddytest/integration/caddyfile_adapt/tls_automation_policies_4.caddyfiletest
+++ b/caddytest/integration/caddyfile_adapt/tls_automation_policies_4.caddyfiletest
@@ -131,8 +131,9 @@ abc.de {
 								"module": "acme"
 							},
 							{
+								"ca": "https://acme.zerossl.com/v2/DV90",
 								"email": "my.email@example.com",
-								"module": "zerossl"
+								"module": "acme"
 							}
 						]
 					}
diff --git a/caddytest/integration/caddyfile_adapt/tls_automation_policies_8.caddyfiletest b/caddytest/integration/caddyfile_adapt/tls_automation_policies_8.caddyfiletest
index 1703178ebc3..bd1bbf2214d 100644
--- a/caddytest/integration/caddyfile_adapt/tls_automation_policies_8.caddyfiletest
+++ b/caddytest/integration/caddyfile_adapt/tls_automation_policies_8.caddyfiletest
@@ -86,8 +86,9 @@ http://localhost:8081 {
 								"module": "acme"
 							},
 							{
+								"ca": "https://acme.zerossl.com/v2/DV90",
 								"email": "abc@example.com",
-								"module": "zerossl"
+								"module": "acme"
 							}
 						]
 					}
diff --git a/caddytest/integration/caddyfile_adapt/tls_automation_policies_global_email_localhost.caddyfiletest b/caddytest/integration/caddyfile_adapt/tls_automation_policies_global_email_localhost.caddyfiletest
index e8ef3a7e968..50fbf51aa83 100644
--- a/caddytest/integration/caddyfile_adapt/tls_automation_policies_global_email_localhost.caddyfiletest
+++ b/caddytest/integration/caddyfile_adapt/tls_automation_policies_global_email_localhost.caddyfiletest
@@ -54,8 +54,9 @@ example.com {
 								"module": "acme"
 							},
 							{
+								"ca": "https://acme.zerossl.com/v2/DV90",
 								"email": "foo@bar",
-								"module": "zerossl"
+								"module": "acme"
 							}
 						]
 					}
diff --git a/caddytest/integration/caddyfile_adapt/tls_dns_ttl.caddyfiletest b/caddytest/integration/caddyfile_adapt/tls_dns_ttl.caddyfiletest
index 02e46763d05..c452bf79f13 100644
--- a/caddytest/integration/caddyfile_adapt/tls_dns_ttl.caddyfiletest
+++ b/caddytest/integration/caddyfile_adapt/tls_dns_ttl.caddyfiletest
@@ -58,14 +58,6 @@ tls {
 									}
 								},
 								"module": "acme"
-							},
-							{
-								"challenges": {
-									"dns": {
-										"ttl": 310000000000
-									}
-								},
-								"module": "zerossl"
 							}
 						]
 					}
diff --git a/caddytest/integration/caddyfile_adapt/tls_explicit_issuer_dns_ttl.caddyfiletest b/caddytest/integration/caddyfile_adapt/tls_explicit_issuer_dns_ttl.caddyfiletest
index 53629e3a1b7..d552599ffe5 100644
--- a/caddytest/integration/caddyfile_adapt/tls_explicit_issuer_dns_ttl.caddyfiletest
+++ b/caddytest/integration/caddyfile_adapt/tls_explicit_issuer_dns_ttl.caddyfiletest
@@ -5,7 +5,7 @@ tls {
 	issuer acme {
 		dns_ttl 5m10s
 	}
-	issuer zerossl {
+	issuer zerossl api_key {
 		dns_ttl 10m20s
 	}
 }
@@ -65,10 +65,9 @@ tls {
 								"module": "acme"
 							},
 							{
-								"challenges": {
-									"dns": {
-										"ttl": 620000000000
-									}
+								"api_key": "api_key",
+								"cname_validation": {
+									"ttl": 620000000000
 								},
 								"module": "zerossl"
 							}
diff --git a/caddytest/integration/caddyfile_adapt/tls_explicit_issuer_propagation_options.caddyfiletest b/caddytest/integration/caddyfile_adapt/tls_explicit_issuer_propagation_options.caddyfiletest
index 032f9284f2b..206d59ca5b5 100644
--- a/caddytest/integration/caddyfile_adapt/tls_explicit_issuer_propagation_options.caddyfiletest
+++ b/caddytest/integration/caddyfile_adapt/tls_explicit_issuer_propagation_options.caddyfiletest
@@ -6,7 +6,7 @@ tls {
 		propagation_delay 5m10s
 		propagation_timeout 10m20s
 	}
-	issuer zerossl {
+	issuer zerossl api_key {
 		propagation_delay 5m30s
 		propagation_timeout -1
 	}
@@ -68,11 +68,10 @@ tls {
 								"module": "acme"
 							},
 							{
-								"challenges": {
-									"dns": {
-										"propagation_delay": 330000000000,
-										"propagation_timeout": -1
-									}
+								"api_key": "api_key",
+								"cname_validation": {
+									"propagation_delay": 330000000000,
+									"propagation_timeout": -1
 								},
 								"module": "zerossl"
 							}
diff --git a/caddytest/integration/caddyfile_adapt/tls_propagation_options.caddyfiletest b/caddytest/integration/caddyfile_adapt/tls_propagation_options.caddyfiletest
index ee4666b667f..43ec9774bcd 100644
--- a/caddytest/integration/caddyfile_adapt/tls_propagation_options.caddyfiletest
+++ b/caddytest/integration/caddyfile_adapt/tls_propagation_options.caddyfiletest
@@ -60,15 +60,6 @@ tls {
 									}
 								},
 								"module": "acme"
-							},
-							{
-								"challenges": {
-									"dns": {
-										"propagation_delay": 310000000000,
-										"propagation_timeout": 620000000000
-									}
-								},
-								"module": "zerossl"
 							}
 						]
 					}
diff --git a/modules/caddyhttp/reverseproxy/fastcgi/client_test.go b/modules/caddyhttp/reverseproxy/fastcgi/client_test.go
index a2227a65332..14a1cf684c7 100644
--- a/modules/caddyhttp/reverseproxy/fastcgi/client_test.go
+++ b/modules/caddyhttp/reverseproxy/fastcgi/client_test.go
@@ -213,8 +213,6 @@ func DisabledTest(t *testing.T) {
 	// TODO: test chunked reader
 	globalt = t
 
-	rand.Seed(time.Now().UTC().UnixNano())
-
 	// server
 	go func() {
 		listener, err := net.Listen("tcp", ipPort)