Repository: caddyserver/caddy

Issue:
Support Encrypted Client Hello (formerly known as ESNI)
Please add support for [TLS Encrypted Client Hello](https://datatracker.ietf.org/doc/draft-ietf-tls-esni/)

- Championed by the EFF: https://www.eff.org/deeplinks/2018/09/esni-privacy-protecting-upgrade-https
- Deployed by Cloudflare: https://blog.cloudflare.com/esni/
- Cloudflare's technical details post: https://blog.cloudflare.com/encrypted-sni/
- Supported by Firefox: https://blog.mozilla.org/security/2018/10/18/encrypted-sni-comes-to-firefox-nightly/
- Supported by NSS: https://bugzilla.mozilla.org/show_bug.cgi?id=1495120
- Work in Progress in picotls: h2o/picotls#155

[Previous ticket](https://github.com/caddyserver/caddy/issues/2327) for ESNI was not recreated after being closed, so here it is.

List taken from [the openssl issue](https://github.com/openssl/openssl/issues/7482).

Discussion:
@francislavoie: https://blog.cloudflare.com/encrypted-client-hello/

I don't think ESNI has a future.

And this isn't actionable for the Caddy project until Golang implements it. And there seems to be no plans to unless there's suddenly wider adoption.@mholt: ECH is its successor though. If Go implements it we should try to support it.@francislavoie: Oh sorry, I totally misread the issue title. My bad! :man_facepalming: @joshxyzhimself: Related

- https://guardianproject.info/2021/11/30/implementing-tls-encrypted-client-hello/
- https://news.ycombinator.com/item?id=29530811@diyism: Any updates? all of nginx,apache,lighttpd have support ECH, how about caddy?
@francislavoie: @diyism I can't find any evidence that they do support it. Where did you see that? Please share links.@diyism: > @diyism I can't find any evidence that they do support it. Where did you see that? Please share links.

I've test it in https://defo.ie/ with msedge dev(https://www.thewindowsclub.com/enable-encrypted-client-hello-in-microsoft-edge):

nginx:  https://draft-13.esni.defo.ie:10413
apache2: https://draft-13.esni.defo.ie:11413
both of the upper two pages show "SSL_ECH_STATUS: success" and a green tick mark.
Can it prove they hava a ECH-ready nginx and apache2 ?@francislavoie: Those are experiments using a custom build of OpenSSL. That's not really "support".

Either way, Caddy doesn't use OpenSSL, it uses Go's stdlib for TLS. So this work doesn't apply to us.@mholt: Yeah, that would be like us building with a custom fork of the Go standard library to support ECH. We could do that ([Cloudflare has one](https://github.com/cloudflare/go)), but given the sensitive nature of the `crypto/tls` package, I would only recommend that for experimental or low-risk environments for now. (CF is surely using it in production, but we don't know how much of the diff is tailored to just their use cases.)

So I guess we can answer "How about Caddy?" with "Sure, go for it" -- just build it with a forked tls package and it is possible. (I'm uncertain if any code changes to Caddy would need to take place, but since you're building from source already, might as well.)

As a side-note, the ECH draft spec seems to have expired: https://datatracker.ietf.org/doc/draft-ietf-tls-esni/@ViViDboarder: I think this doesn’t change anything as Caddy’s stance appears to be “when it’s supported in Go stdlib,” which makes total sense to me, but it looks like ECH has legs and is now supported in mainline Firefox: https://blog.mozilla.org/en/products/firefox/encrypted-hello/@mholt: Yeah; to clarify, we like ECH, and will support it someday assuming Go's standard lib implements it. (Because we're wary about switching to a fork of crypto/tls for our standard distribution of Caddy.)

However, I should note that ECH has limited practical use currently as DNS lookups still reveal hostnames in plaintext most of the time, and packets still have IP addresses that aren't encrypted.

It's just really, really hard to solve the problem people are hoping to solve, and ECH is not the whole solution, and without the whole solution, users are still at risk.

But it's cool I guess. A step in the right direction.@ViViDboarder: Firefox also defaults to DNS over TLS as well, so this is another step to close the gaps.

IP addresses is a big gap. Though it’s one that can be mitigated, depending on trade offs one is willing to make. I like to self host, so I run Caddy on my own server with my IP. So my connection is going to be known that way, but others, who may not care about their own end to end control, can run their server behind a CDN, like Cloudflare. In that case, the IP revealed is one that gives an interloper very little information.

I’m glad to have seen more and more progress over the last few years and excited to see what’s to come!@dennisjackson: @mholt - Just to clarify, ECH is the last part of the solution. There were four places where website names were leaking in plaintext: 
 - In the TLS Server Certificate (fixed by TLS 1.3) 
 - In the DNS lookup (fixed by DNS-over-HTTPS - which has already shipped in Firefox) 
 - In the certificate revocation checking (fixed by OCSP Stapling and CRLite)
 - In the SNI field (fixed by ECH). 
 
On the topic of IPs, they are far less unique than people realise. Cloudflare have already experimented with serving all of their websites from a single IP address [1]. In their measurement study, over 50% of the entire .com, .net, .info and .org namespace (255 million websites) is available from only ~100 IPs. This is obviously not great from a centralisation perspective, but that seems like more a reason for Caddy to support privacy features like ECH, not less!@mholt: > Firefox also defaults to DNS over TLS as well, so this is another step to close the gaps.

I'm curious how widespread this is; if it's mostly us tech geeks who actually use this feature -- I know, for example, my parents don't benefit from this. I'd still wager the vast majority of DNS lookups are cleartext.

And yeah, IP addresses in _some_ cases don't reveal much, inasmuch as we centralize to a single hosting provider.

>  but that seems like more a reason for Caddy to support privacy features like ECH, not less!

To AGAIN clarify, we like ECH. I just want to make sure that people understand that it is only a part of the solution and the solution is still largely incomplete. Your sites and web browsing experience may already have the whole solution, but (without seeing stats I can only guess) I'd wager that most of the Internet still has a long way to go, so I want to make sure we don't hype up something that isn't real for most people yet.@dennisjackson: > I'm curious how widespread this is; if it's mostly us tech geeks who actually use this feature -- I know, for example, my parents don't benefit from this. I'd still wager the vast majority of DNS lookups are cleartext.

It's the [default](https://support.mozilla.org/en-US/kb/dns-over-https-doh-faqs#w_more-about-firefoxs-implementation-of-dns-over-https) out of the box in North America, Russia and Ukraine at the moment. ECH is also enabled by default worldwide. 
 
> To AGAIN clarify, we like ECH.

That's great to hear!  

> I just want to make sure that people understand that it is only a part of the solution and the solution is still largely incomplete. 

One step at a time :-). 


@mholt: (What I'm saying is that most people don't use Firefox, unfortunately. I know other browsers plan to roll it out too, but we're not there yet.)@Harusakii: > (What I'm saying is that most people don't use Firefox, unfortunately. I know other browsers plan to roll it out too, but we're not there yet.)

I think we are there now, Chrome released the update 117.0.5938.150 which now enables ECH on default. Just saying though ^^@networkException: Where did you see that? Checking out chromium source at the `117.0.5938.150` tag declares `kEncryptedClientHello` as `FEATURE_DISABLED_BY_DEFAULT` in `net/base/features.cc`@Harusakii: Idk about Chromium but as of https://chromestatus.com/roadmap it is in Origin Trial/Prepare to Ship for 117 so it will come later, but either way Chrome did enable it by default (you may need to reset your chrome://flags) and then you can check by going to https://www.cloudflare.com/ssl/encrypted-sni to see it enabled@mohammed90: We're dependent on Go standard library to support this function. Watch golang/go#63369. It seems to be going in a positive direction.@rugk: Firefox now supports ECH, would be great if you could support it, too: https://blog.mozilla.org/en/products/firefox/encrypted-hello/@francislavoie: @rugk as it's been said multiple times in this thread, there's nothing we (the Caddy team) can do until the Go stdlib has support for ECH.@PrivacyOsint2: @francislavoie @mholt @mohammed90 @Gunni 

They added support for ECH.

crypto/tls: add ech client support

https://github.com/golang/go/commit/9eeb627f606e713854e117dd4e52af5bcad28b66@PrivacyOsint2:  crypto/tls: support Encrypted Client Hello in clients #63369 
 Closed / Added
 
https://github.com/golang/go/issues/63369
@Gunni: > @francislavoie @mholt @mohammed90 @Gunni 
> 
> They added support for ECH.
> 
> crypto/tls: add ech client support
> 
> https://github.com/golang/go/commit/9eeb627f606e713854e117dd4e52af5bcad28b66

Only client side logic https://github.com/golang/go/issues/63369#issuecomment-2126161985@coolaj86: Status has been updated to Accepted: https://github.com/golang/go/issues/68500

Release targeted for Go 1.24: https://github.com/golang/go/issues/63369#issuecomment-2083070008 \
(for reference, the current version is 1.23.1 and 1.24 is expected to land around February 2025 )@mholt: As I design this, my current understanding of ECH is, based on the [ESNI draft 22](https://www.ietf.org/archive/id/draft-ietf-tls-esni-22.html#name-overview):

> A client-facing server enables ECH by publishing an ECH configuration, which is an encryption public key and associated metadata. The server must publish this for all the domains it serves via Shared or Split Mode.

So:

- At least one key pair, but possibly others, are generated by the server (or admin -- but let's be real, we're fond of automation here).

On publishing keys:

> This document defines the ECH configuration's format, but delegates DNS publication details to [[RFC9460](https://www.ietf.org/archive/id/draft-ietf-tls-esni-22.html#RFC9460)]. See [[ECH-IN-DNS](https://www.ietf.org/archive/id/draft-ietf-tls-esni-22.html#ECH-IN-DNS)] for specifics about how ECH configurations are advertised in HTTPS records. Other delivery mechanisms are also possible. For example, the client may have the ECH configuration preconfigured.

- The public key is published, usually to DNS records (!) ... but also may be preconfigured into clients (so, kinda like SSH in a way).
- The relevant draft RFC with details for servers is here: https://datatracker.ietf.org/doc/html/draft-ietf-tls-svcb-ech-05#name-server-behavior
- There's also RFC 9460 about the HTTPS RRs: https://www.rfc-editor.org/rfc/rfc9460#https

The Go standard library exposes one property for ECH servers, [`EncryptedClientHelloKeys`](https://pkg.go.dev/crypto/tls#Config.EncryptedClientHelloKeys). The tricky thing is it takes a `Config []byte`, which is the already-serialized form of the ECHConfig struct defined by the ESNI draft spec. There is no standard function that produces this serialization... yet. There is [this package](https://github.com/OmarTariq612/goech) which seems to do this, but it'd be nice if we didn't have new dependencies for this.

Anyway... bringing this all down to our task here to integrate this capability into Caddy:

To make this as easy, and as least error-prone as possible, for Caddy users, I suppose we could expose a config object for ECH in the TLS connection policies that enable ECH. The user would have to either provide keys or have Caddy automate them. Of course, we'll prefer the latter.

We have the unique opportunity to do something really special here, with the [`libdns` packages](https://github.com/libdns) we already use for ACME DNS challenges and things like the [caddy-dynamicdns plugin](https://github.com/mholt/caddy-dynamicdns). With ECH enabled, and DNS provider credentials in the config, we can generate ECH keys, publish them to DNS records, and serve ECH with regularly-rotated keys. i.e. this could be really easy when it comes to user-facing config.

Any thoughts before I take this further? Am I way off base here?@christaikobo: Apologies in advance if my question sounds stupid here.

The cloudflare implementation seems to have an outer and inner SNI. I'm
wondering if that is compulsory and what plan caddy has regarding this
issue.

On Tue, Feb 11, 2025, 21:21 Matt Holt ***@***.***> wrote:

> As I design this, my current understanding of ECH is, based on the ESNI
> draft 22
> <https://www.ietf.org/archive/id/draft-ietf-tls-esni-22.html#name-overview>
> :
>
> A client-facing server enables ECH by publishing an ECH configuration,
> which is an encryption public key and associated metadata. The server must
> publish this for all the domains it serves via Shared or Split Mode.
>
> So:
>
>    - At least one key pair, but possibly others, are generated by the
>    server (or admin -- but let's be real, we're fond of automation here).
>
> On publishing keys:
>
> This document defines the ECH configuration's format, but delegates DNS
> publication details to [RFC9460
> <https://www.ietf.org/archive/id/draft-ietf-tls-esni-22.html#RFC9460>].
> See [ECH-IN-DNS
> <https://www.ietf.org/archive/id/draft-ietf-tls-esni-22.html#ECH-IN-DNS>]
> for specifics about how ECH configurations are advertised in HTTPS records.
> Other delivery mechanisms are also possible. For example, the client may
> have the ECH configuration preconfigured.
>
>
>    - The public key is published, usually to DNS records (!) ... but also
>    may be preconfigured into clients (so, kinda like SSH in a way).
>    - The relevant draft RFC with details for servers is here:
>    https://datatracker.ietf.org/doc/html/draft-ietf-tls-svcb-ech-05#name-server-behavior
>    - There's also RFC 9460 about the HTTPS RRs:
>    https://www.rfc-editor.org/rfc/rfc9460#https
>
> The Go standard library exposes one property for ECH servers,
> EncryptedClientHelloKeys
> <https://pkg.go.dev/crypto/tls#Config.EncryptedClientHelloKeys>. The
> tricky thing is it takes a Config []byte, which is the already-serialized
> form of the ECHConfig struct defined by the ESNI draft spec. There is no
> standard function that produces this serialization... yet. There is this
> package <https://github.com/OmarTariq612/goech> which seems to do this,
> but it'd be nice if we didn't have new dependencies for this.
>
> Anyway... bringing this all down to our task here to integrate this
> capability into Caddy:
>
> To make this as easy, and as least error-prone as possible, for Caddy
> users, I suppose we could expose a config object for ECH in the TLS
> connection policies that enable ECH. The user would have to either provide
> keys or have Caddy automate them. Of course, we'll prefer the latter.
>
> We have the unique opportunity to do something really special here, with
> the libdns packages <https://github.com/libdns> we already use for ACME
> DNS challenges and things like the caddy-dynamicdns plugin
> <https://github.com/mholt/caddy-dynamicdns>. With ECH enabled, and DNS
> provider credentials in the config, we can generate ECH keys, publish them
> to DNS records, and serve ECH with regularly-rotated keys. i.e. this could
> be really easy when it comes to user-facing config.
>
> Any thoughts before I take this further? Am I way off base here?
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/caddyserver/caddy/issues/4221#issuecomment-2652692817>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AFY26LEOWYPTVLWBZS7ZZLL2PLK67AVCNFSM6AAAAABJMB5KJKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMNJSGY4TEOBRG4>
> .
> You are receiving this because you are subscribed to this thread.Message
> ID: ***@***.***>
> [image: mholt]*mholt* left a comment (caddyserver/caddy#4221)
> <https://github.com/caddyserver/caddy/issues/4221#issuecomment-2652692817>
>
> As I design this, my current understanding of ECH is, based on the ESNI
> draft 22
> <https://www.ietf.org/archive/id/draft-ietf-tls-esni-22.html#name-overview>
> :
>
> A client-facing server enables ECH by publishing an ECH configuration,
> which is an encryption public key and associated metadata. The server must
> publish this for all the domains it serves via Shared or Split Mode.
>
> So:
>
>    - At least one key pair, but possibly others, are generated by the
>    server (or admin -- but let's be real, we're fond of automation here).
>
> On publishing keys:
>
> This document defines the ECH configuration's format, but delegates DNS
> publication details to [RFC9460
> <https://www.ietf.org/archive/id/draft-ietf-tls-esni-22.html#RFC9460>].
> See [ECH-IN-DNS
> <https://www.ietf.org/archive/id/draft-ietf-tls-esni-22.html#ECH-IN-DNS>]
> for specifics about how ECH configurations are advertised in HTTPS records.
> Other delivery mechanisms are also possible. For example, the client may
> have the ECH configuration preconfigured.
>
>
>    - The public key is published, usually to DNS records (!) ... but also
>    may be preconfigured into clients (so, kinda like SSH in a way).
>    - The relevant draft RFC with details for servers is here:
>    https://datatracker.ietf.org/doc/html/draft-ietf-tls-svcb-ech-05#name-server-behavior
>    - There's also RFC 9460 about the HTTPS RRs:
>    https://www.rfc-editor.org/rfc/rfc9460#https
>
> The Go standard library exposes one property for ECH servers,
> EncryptedClientHelloKeys
> <https://pkg.go.dev/crypto/tls#Config.EncryptedClientHelloKeys>. The
> tricky thing is it takes a Config []byte, which is the already-serialized
> form of the ECHConfig struct defined by the ESNI draft spec. There is no
> standard function that produces this serialization... yet. There is this
> package <https://github.com/OmarTariq612/goech> which seems to do this,
> but it'd be nice if we didn't have new dependencies for this.
>
> Anyway... bringing this all down to our task here to integrate this
> capability into Caddy:
>
> To make this as easy, and as least error-prone as possible, for Caddy
> users, I suppose we could expose a config object for ECH in the TLS
> connection policies that enable ECH. The user would have to either provide
> keys or have Caddy automate them. Of course, we'll prefer the latter.
>
> We have the unique opportunity to do something really special here, with
> the libdns packages <https://github.com/libdns> we already use for ACME
> DNS challenges and things like the caddy-dynamicdns plugin
> <https://github.com/mholt/caddy-dynamicdns>. With ECH enabled, and DNS
> provider credentials in the config, we can generate ECH keys, publish them
> to DNS records, and serve ECH with regularly-rotated keys. i.e. this could
> be really easy when it comes to user-facing config.
>
> Any thoughts before I take this further? Am I way off base here?
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/caddyserver/caddy/issues/4221#issuecomment-2652692817>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AFY26LEOWYPTVLWBZS7ZZLL2PLK67AVCNFSM6AAAAABJMB5KJKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMNJSGY4TEOBRG4>
> .
> You are receiving this because you are subscribed to this thread.Message
> ID: ***@***.***>
>
@devhaozi: > The Go standard library exposes one property for ECH servers, [`EncryptedClientHelloKeys`](https://pkg.go.dev/crypto/tls#Config.EncryptedClientHelloKeys). The tricky thing is it takes a `Config []byte`, which is the already-serialized form of the ECHConfig struct defined by the ESNI draft spec. There is no standard function that produces this serialization... yet. There is [this package](https://github.com/OmarTariq612/goech) which seems to do this, but it'd be nice if we didn't have new dependencies for this.

This package https://github.com/c2FmZQ/ech also can use to generate ECH config.@christaikobo: Another question: (please ignore it if it is too early to consider)

How will it interact with the layer4 addon? More specifically, the SNI mathers?@mholt: I got it working!

It's currently very spikey, but I just needed to prove my understanding of things and show that it works. I will be refactoring and refining it, and looking at addressing the security considerations in the RFC as soon as possible.

Config example (will be changing, but should remain about this simple):

![Image](https://github.com/user-attachments/assets/d59db025-8dc1-4517-8b35-4c04a48a2db8)

I'll likely hoist this to a global config setting, since I imagine most users will want a single outer domain that hides all their sites, right?

That config causes Caddy to provision an HTTPS-type DNS record with the `ech` SvcParamKey containing the public key and the outer SNI. Caddy then manages a certificate for the site's domain as usual, and also the outer SNI.

Then in Firefox with DoH (DNS-over-HTTPS) enabled, I loaded the domain name of the site I was trying to access:

![Image](https://github.com/user-attachments/assets/33153de2-23dc-44c9-b594-32eaf5133d76)

And sure enough it loaded. But was it hidden? Wireshark says yes:

![Image](https://github.com/user-attachments/assets/60ab832a-1586-4627-998a-73303669ad70)

Only the outer SNI is visible on the wire.

---

There's still a lot to consider. I have no idea yet if I'm doing it _properly_ other than it seems to be working. I will study the RFC more.

---

@christaikobo 

> The cloudflare implementation seems to have an outer and inner SNI. I'm wondering if that is compulsory and what plan caddy has regarding this issue.

I think it is... at least as much as SNI is "compulsory" without ECH. The outer SNI is needed (?) so a visible handshake can be completed and a certificate served. The inner SNI is the actual hostname being accessed.

> How will it interact with the layer4 addon? More specifically, the SNI mathers?

Not quite sure yet... layer4 will need to be updated to support ECH. The ECH spec defines two modes: "shared" and "split". In shared mode, the server both unwraps the inner SNI, and terminates TLS. In split mode, the server only unwraps the inner SNI, and proxies the "real" TLS handshake to a backend without terminating it. (Pretty cool IMO, although layer4 already supports bespoke TLS-in-TLS tunnels for any clients that happen to support that.)

@devhaozi 

> This package https://github.com/c2FmZQ/ech also can use to generate ECH config.

Thanks a lot for this; actually the package I found doesn't seem to marshal the ECHConfig correctly, but that package does... I likely won't be using either, since the actual borrowed code is only about 15 lines long, and both packages had a nearly identical implementation (which makes sense, given that there's only 1 proper way to do it.)@devhaozi: > I think it is... at least as much as SNI is "compulsory" without ECH. The outer SNI is needed (?) so a visible handshake can be completed and a certificate served. The inner SNI is the actual hostname being accessed.

From my understanding, outer SNI is required, but it should be customizable to be different from inner SNI. (In [Cloudflare's implementation](https://developers.cloudflare.com/ssl/edge-certificates/ech/) they use `cloudflare-ech.com` as outer SNI)@mholt: I figured out what was wrong with the package at https://github.com/OmarTariq612/goech -- it marshals an ECHConfig as an ECHConfigList (with 1 ECHConfig in it), and the MarshalBinary for ECHConfigList, I think, improperly encodes a single length at the beginning instead of a length per element.

Anyway, I have made my own implementation that Caddy will use. The HTTPS DNS record takes a base64-encoded ECHConfigList and the Go crypto/tls package takes one or more binary-encoded ECHConfig (paired with the private keys).

I have that all working now.

It seems that a certificate is _not_ required for the outer SNI, which is convenient.

The remaining task is to figure out how to design the Caddy config.@christaikobo: > It seems that a certificate is _not_ required for the outer SNI, which is convenient.

Interesting, does this mean we will be allowed to use any domain as outer SNI?
@Gunni: What about not using an outer SNI? Is that permissible? Or spoofing any outer name if we aren't serving it anyway? @christaikobo: > What about not using an outer SNI? Is that permissible? Or spoofing any outer name if we aren't serving it anyway?

Yes this is pretty much what is going through my mind. For large cooperation it makes sense to have an outer SNI in order to hide an inner SNI. For smaller organization and individuals with only 1 or a couple of domains, the effectiveness is drastically reduced. 

For example, hiding `blog.mydomain.com` with `ech.mydomain.com` is pretty much useless if `mydomain.com` is the only domain I have. @mholt: @christaikobo and @Gunni 

> Interesting, does this mean we will be allowed to use any domain as outer SNI?

> What about not using an outer SNI? Is that permissible? Or spoofing any outer name if we aren't serving it anyway?

No; to clarify, a certificate for the outer/public name is optional, but is recommended. If the server cannot decrypt the client's ECH extension (e.g. old or wrong config presented), then the server can complete the handshake with the outer ClientHello (thus it would need a certificate) so the client can safely get the updated configs to retry (https://www.ietf.org/archive/id/draft-ietf-tls-esni-22.html#rejected-ech). Basically, the outer SNI must be controlled by (or delegated to) the ECH server for successful, tracking-resilient client retries.

https://www.ietf.org/archive/id/draft-ietf-tls-esni-22.html#section-8.1.1:

> public_name: The DNS name of the client-facing server, i.e., the entity trusted to update the ECH configuration. This is used to correct misconfigured clients, as described in [Section 6.1.6](https://www.ietf.org/archive/id/draft-ietf-tls-esni-22.html#rejected-ech).
> ...
> The retry mechanism repairs inconsistencies, provided the server is authoritative for the public name. ... Provided the server can present a certificate valid for the public name, the client can safely retry with updated settings, as described in [Section 6.1.6](https://www.ietf.org/archive/id/draft-ietf-tls-esni-22.html#rejected-ech).
> ..
> In order to ensure that the retry mechanism works successfully servers SHOULD ensure that every endpoint which might receive a TLS connection is provisioned with an appropriate certificate for the public name. This is especially important during periods of server reconfiguration when different endpoints might have different configurations.

Inconsistencies are possible as the latest DNS records are not always visible to clients, or whatever other method of publication is being employed; the point is that the client and the server both need to have =exact byte-for-byte the-same ECHConfigs. If they don't, my understanding is that the client can use a retry mechanism to resolve inconsistencies. However, doing this in a way that doesn't defeat the purpose of ECH requires the origin server to provide a certificate for the public name.

See also: https://www.ietf.org/archive/id/draft-ietf-tls-esni-22.html#name-authenticating-for-the-publ

**In summary:** No, technically, an ECH server does NOT need a certificate for the public name. However, this can make client reliability more brittle, since it will not be able to safely resolve inconsistencies / update its state. Thus, the server SHOULD be authoritative for that public name. Caddy will still manage a certificate for the public name, but maybe there'll be an option to disable that requirement. It can still serve ECH to clients while it gets its first certificate for the public name.

---

> For large cooperation it makes sense to have an outer SNI in order to hide an inner SNI. For smaller organization and individuals with only 1 or a couple of domains, the effectiveness is drastically reduced.
>
> For example, hiding blog.mydomain.com with ech.mydomain.com is pretty much useless if mydomain.com is the only domain I have.

True. A DNS scan can reveal which "hidden" names are using ECH and what their corresponding public names are. The spec also says:

> Usage of this mechanism reveals that a client is connecting to a particular service provider, but does not reveal which server from the anonymity set terminates the connection.
> ...
> ECH is not in itself sufficient to protect the identity of the server. The target domain may also be visible through other channels, such as plaintext client DNS queries or visible server IP addresses. However, DoH [[RFC8484](https://www.ietf.org/archive/id/draft-ietf-tls-esni-22.html#RFC8484)] and DPRIVE [[RFC7858](https://www.ietf.org/archive/id/draft-ietf-tls-esni-22.html#RFC7858)] [[RFC8094](https://www.ietf.org/archive/id/draft-ietf-tls-esni-22.html#RFC8094)] provide mechanisms for clients to conceal DNS lookups from network inspection, and many TLS servers host multiple domains on the same IP address. Private origins may also be deployed behind a common provider, such as a reverse proxy. In such environments, the SNI remains the primary explicit signal used to determine the server's identity.

This is actually one valid criticism of ECH: it tends to favor centralization; i.e. bigcorps like Cloudflare/Google/Amazon hosting all the things, since the more things that are hidden behind a central authority, the more anonymous/ambiguous the visible parts become.@christaikobo: Thanks for the elaborated explanation, very informative. 

I was wrong to get my hopes a little bit too high. But if caddy decides to ship with the option for disabling the requirement of a public certification, I will play around a bit and see how it affects things. Maybe it's a compromise some of us are willing to make, reduced reliability in exchange for better anonymity. @mholt: I'm not sure not having a public name affords you "better anonymity," and I'm worried that this might be a common and slippery/dangerous mistake that a lot of people assume. I believe, the reality is, public name does not practically reveal more information than the IP address being connected to.

The spec refers to an **anonymity set** as this:

> Co-located servers with consistent externally visible TLS configurations and behavior, including supported versions and cipher suites and how they respond to incoming client connections, form an anonymity set.

So, if you're on Cloudflare, Google, Amazon, DigitalOcean, etc., then your anonymity set, in theory, could be quite large. If you are self-hosting on your home Internet service and relying on ECH for anonymity, your anonymity set is drastically smaller, regardless of whether a public name is used or not, by the mere fact that a set is comprised of _co-located servers_.

The spec advises making your anonymity set as large as possible:

> With regards to handshake privacy, client-facing server configuration determines the size of the anonymity set. For example, if a client-facing server uses distinct ECHConfig values for each host, then each anonymity set has size k = 1. Client-facing servers SHOULD deploy ECH in such a way so as to maximize the size of the anonymity set where possible. This means client-facing servers should use the same ECHConfig for as many hosts as possible.

(In terms of anonymity, there should be as few ECHConfigs as possible on the Internet; which of course calls up the criticism that ECH is advocating centralization, but I digress.)

A public name does not affect these factors at all, and only improves the reliability of clients being able to connect. So, I don't think specifying a public name in the config affects the anonymity of the setup.

In other words, based on my current limited understanding, using a public name is orthogonal to anonymity. Sure, don't choose a name like `ech-for-hidden-service.com`, but even a random domain name would do fine.

Remember, too, that hidden names are still publicly known (assuming the public Internet & DNS), since the HTTPS-type record that will conventionally be used to publish ECH configs are also public and can be determined with a very simple DNS lookup. So the fact that a name is being hidden is public information regardless. (You could, theoretically, hide a name without everyone knowing it is hidden by publishing ECH configs privately, but that's... obviously not publicly useful.)

So, what about specifying a public name materially reduces anonymity?@christaikobo: I am thinking about SNI based censorship (and/or just snooping), especially (but not limited to) a whitelist approach. Masquerading as `cloudflare-ech.com` (or simply without any SNI at all) would get you pass a lot (not all) of these. Most firewall system are automated and they are not going to conduct targeted and sophisticated investigation, so we don't need absolute anonymity, we just need enough of it to get by. (IMHO the thing about privacy and anonymity is always a trade off, you can never achieve absolute safety in that regard)

What you have talked about are undeniably true, in a relatively ideal world where most people play by some basic rules. Unfortunately it is not always the case. 

I know some of what I just said might be controversial, but I want to say leaving the option open is definitely going to be useful for some of the users, especially if it is not contradictory with the RFC itself. Nobody is misleading anybody that there is a mirage of absolute privacy and anonymity with ECH. @mholt: I guess I don't understand why omitting a public name would help you. The censor would then just block handshakes advertising ECH that lack SNI.@christaikobo: That's why I was interested in masquerading as cloudflare-ech.com.

Another factor to consider is snooping. 

At the end of the day, without compromising other aspects of the feature, the ADDED possible benefit (that is also in line with the RFC) wouldn't hurt anyone. Yes censor could react and tighten the control, but it is possible that it won't. Maybe they are worried about false positive, maybe they are worried about complaints, etc.. @shawnz: > I guess I don't understand why omitting a public name would help you. The censor would then just block handshakes advertising ECH that lack SNI.

If a provider wanted to indiscriminately block ECH, why would they care about whether there's a public name present or not? 

GREASE is the mechanism by which you can discourage providers from indiscriminately blocking ECH, and it doesn't require a public name to be present.@mholt: > That's why I was interested in masquerading as cloudflare-ech.com.

Wouldn't work, because your IP is not allocated to Cloudflare.

> Another factor to consider is snooping.

What about it though? You haven't presented a tangible explanation as to why omitting a public SNI enhances privacy more than other alternatives.

> If a provider wanted to indiscriminately block ECH, why would they care about whether there's a public name present or not?

We're not talking about indiscriminately blocking of ECH, but rather blocking of individual ECH configs. An empty string is just the same as a non-empty string in terms of blocking things.@shawnz: Sure, but what's the motivation to block that config? It's no different than a server owner having no server name and a GREASEd ECH block, which doesn't seem especially suspicious to me and could be a common configuration in practice (although I don't have the numbers to back that up)@christaikobo: When I said snooping, I meant the ISP recording all SNI of traffic from and
to a user.

And why does the server IP has to be allocated to cloudflare? Are you
talking about it wouldn't work at all as an ECH setting? Or are you
suggesting that since the server IP doesn't belong to cloudflare, the outer
SNI spoofing is easy to detect?

On Sat, Feb 15, 2025, 16:09 Matt Holt ***@***.***> wrote:

> That's why I was interested in masquerading as cloudflare-ech.com.
>
> Wouldn't work, because your IP is not allocated to Cloudflare.
>
> Another factor to consider is snooping.
>
> What about it though? You haven't presented a tangible explanation as to
> why omitting a public SNI enhances privacy more than other alternatives.
>
> If a provider wanted to indiscriminately block ECH, why would they care
> about whether there's a public name present or not?
>
> We're not talking about indiscriminately blocking of ECH, but rather
> blocking of individual ECH configs. An empty string is just as easy to
> block as a non empty SNI string.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/caddyserver/caddy/issues/4221#issuecomment-2661156408>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AFY26LE2KUJ2RFTXZPWPAMT2P7JKNAVCNFSM6AAAAABJMB5KJKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMNRRGE2TMNBQHA>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
> [image: mholt]*mholt* left a comment (caddyserver/caddy#4221)
> <https://github.com/caddyserver/caddy/issues/4221#issuecomment-2661156408>
>
> That's why I was interested in masquerading as cloudflare-ech.com.
>
> Wouldn't work, because your IP is not allocated to Cloudflare.
>
> Another factor to consider is snooping.
>
> What about it though? You haven't presented a tangible explanation as to
> why omitting a public SNI enhances privacy more than other alternatives.
>
> If a provider wanted to indiscriminately block ECH, why would they care
> about whether there's a public name present or not?
>
> We're not talking about indiscriminately blocking of ECH, but rather
> blocking of individual ECH configs. An empty string is just as easy to
> block as a non empty SNI string.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/caddyserver/caddy/issues/4221#issuecomment-2661156408>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AFY26LE2KUJ2RFTXZPWPAMT2P7JKNAVCNFSM6AAAAABJMB5KJKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMNRRGE2TMNBQHA>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>
@mholt: > Sure, but what's the motivation to block that config? It's no different than a server owner having no server name and a GREASEd ECH block, which doesn't seem especially suspicious to me and could be a common configuration in practice (although I don't have the numbers to back that up)

I think that's my point, an empty outer server name isn't really beneficial.

> When I said snooping, I meant the ISP recording all SNI of traffic from and to a user.

So? Why does the public SNI reveal something secret or private to the ISP?

> And why does the server IP has to be allocated to cloudflare?

The second one:

> Or are you suggesting that since the server IP doesn't belong to cloudflare, the outer SNI spoofing is easy to detect?

It is obvious you aren't connecting to a Cloudflare service.

I still am not seeing a compelling reason to completely omit an outer SNI.

@christaikobo: > So? Why does the public SNI reveal something secret or private to the ISP?

I thought this is the whole point of privacy? There are certain things that I DO NOT want to share: The service I am hosting when I am the server, the service I am visiting when I am the user. By extension then the whole idea of ECH is useless, because Why does the SNI reveal something secret or private to the ISP? I have said earlier that for small organization and individual, hiding private SNI with public SNI is pretty much useless, because you are revealing an SNI anyway and if you only have one domain, it doesn't really matter if you are revealing "a.domain.com" or "b.domain.com".

> It is obvious you aren't connecting to a Cloudflare service.

I have been dealing with the world's largest and most powerful censorship system for a decade now. I am not an expert on the technical side of things, far from it. But I do have a lot of practical experience, and I can say with 100% confidence that large scale censorship system will not deploy the semi targeted investigation you have alluded to, it is difficult to automate, error prone, leads to significant false positive, and most important of all, is an endless game of whack-a-mole. Who says I HAVE TO masquerade as `cloudflare-ech.com`? There are probably hundreds of different CDNs around the world. I am not sure if you heard of PCDN or not, but it is very popular in certain countries where residential bandwidth is far cheaper than commercial datacenter bandwidth. In those countries, it is very common to have users to connect to CDN with residential IPs. With that information, your statement does not stand very strong. 

Or maybe I don't need to masquerade as a CDN at all, just switch my blocked domain with another innocent obscure domain as SNI. Obviously I don't have certificate for it, but that is not a threat to the user or the legitimate owner of that obscure domain, for apparent reasons.  

The point is, the idea of the masquerade being too obvious is not reflective of the real situation at all. I am not implying that caddy with ECH enabled can defeat THE censorship system I just mentioned, I am merely using it as an example to illustrate the challenges a typical censorship system (maybe an ISP level or cooperate level censorship instead of state level) faces, challenges that their adversaries can exploit. 

Maybe you are not seeing the compelling reason because you haven't been in our shoes. I want to reiterate that if it is potentially useful to a not-insignificant number of users, and it is conformant to the RFC, there is no reason to not provide at least the option for the users. (Unless it demands too much development resources, then it's a different topic) 

Also just to be clear my stance is that I want to masquerade the outer SNI, but if that is impossible I can compromise with an omitted SNI. If neither is impossible, then ECH in caddy is just plain useless for me personally. 
@shawnz: > I think that's my point, an empty outer server name isn't really beneficial.

Well, to be clear I'm not necessarily convinced of the benefits of allowing no outer SNI either, I'm just saying that I don't think there's any incentive for a provider to block that config like you suggested earlier.

But I could think of some niche benefits of allowing it. If the provider blocks you by IP address, you could just get a new IP for your instance without having to change your domain name (which could invalidate links and bookmarks, require software to be redeployed, etc). But if you spoof the outer SNI, that provides opportunities for easy detection that something suspicious is going on. So having no outer SNI might really be the best option for some people in this situation.

EDIT: Although, I guess you could just change the domain name of the outer SNI and keep the inner SNI on the blocked domain.@Helanzy: I want to provide a different point of view: will ECH omitting public SNI be viewed as non-HTTPS connection? 

The scenario in this POV is a bit different: some ISPs forbids HTTP/HTTPs to be hosted on residential IP (home user's network, e.g. expose homelab/NAS services for Internet access). In this case the existence of HTTP/HTTPS is based on the domain sniffed out from HTTP or TLS SNI.

ESNI may help this issue by encrypting the SNI part. But outer SNI in ECH is designed to make the handshake look like normal HTTPS connection as much as possible, to mitigate the impact of network ossification, as stated in [Cloudflare's blog](https://blog.cloudflare.com/encrypted-client-hello/#the-spectre-of-ossification). So ECH with outer SNI plays no difference in this scenario.

I am imaging utilizing ECH in this way to solve the problem:
1.  Server hosted on residential IP on non-443 port, serving only ECH connection, without outer SNI. All connections not using ECH are rejected, to prevent non-ECH HTTPS client from accessing the site.
2. Client access to the site can be accomplished in two ways (in either way outer SNI in `HTTPS` record is empty):
   1. Only client supporting ECH can access the site.
   2. Making use of DNS `HTTPS/A/AAAA` record. Inner domain's `A/AAAA` records are resolved to a data center IP that relays connection to server on residential IP using ECH (sounds like [split mode mentioned in ECH RFC](https://www.ietf.org/archive/id/draft-ietf-tls-esni-22.html#name-overview)), while `HTTPS` record is resolved to residential IP using `ipv4hint/ipv6hint` + `port` field. In this way, clients can fallback to use data center IP to access the site if they do not support ECH.

If ECH without outer SNI looks exactly the same as simple TLS ClientHello without SNI, this would do the job. Is there any information in ECH's ClientHello to makes it different with existing TLS connections and thus can be detected?@mholt: @christaikobo 

Sorry if I seem dismissive -- I want to be clear that I value your feedback and experience. I'm trying to _understand_ it, and for me, that understanding comes by way of scrutiny. It's how I challenge my current view of the topic. So please forgive me if I break it down, I really am just trying to make sure I am on the same page, since I don't believe there's much published literature on this topic yet. (I haven't found it if there is.) So I feel like we're forging ahead here.

> I have said earlier that for small organization and individual, hiding private SNI with public SNI is pretty much useless, because you are revealing an SNI anyway and if you only have one domain, it doesn't really matter if you are revealing "a.domain.com" or "b.domain.com".

If the outer SNI is not that of the services in the anonymity set, then what is the harm in it?

Also, can you help me understand why "if you only have one domain" is a relevant constraint? I don't think there's any reason you have to only have 1 domain. Do you mean _registered domain_?Even if you do only have one of those, maybe for cost reasons, you could serve the actual service at `service.example.com` but have the public name be `example.com` -- something generic. Or maybe `ech.example.com`. What does that reveal that can't already be inferred from casual inspection (e.g. IP address etc)?

> there is no reason to not provide at least the option for the users.

I don't _think_ it will be technically challenging or development-intensive to provide the option (I'll know better a little further along), but the only reason I'm opposed to just doing this without a really compelling reason is because I'm worried about introducing security/privacy _theater_ into the application. If people think they have more privacy without a public name, and that's not provably true, then that's harmful because it makes the protocol and the whole ecosystem more brittle. Clients will fail to connect entirely -- or worse, resort to insecure methods of connecting as a last resort -- without a public SNI.

Currently, my understanding is that:

- The public name is technically orthogonal to privacy relevant to ECH (according to the scope outlined in the spec).
- An empty public name is technically/functionally equivalent to a public name for which you are not the authoritative server: you cannot get a certificate for it, so retries aren't available.
- An empty public name is no less revealing than a non-empty public name, assuming the non-empty public name is generic (i.e. not related to the services being hidden).

@shawnz 

> I'm just saying that I don't think there's any incentive for a provider to block that config like you suggested earlier.

we've seen things like this before, a "mass casualty" event as it were, where an entire IP subnet will be blocked even though only a specific site or two was being targeted. The site changed IP addresses often enough that the powers that be deemed it better to just block the whole range (temporarily, I think). ECH raises the costs of such censorship, but it's not always too high.

> But I could think of some niche benefits of allowing it. If the provider blocks you by IP address, you could just get a new IP for your instance without having to change your domain name (which could invalidate links and bookmarks, require software to be redeployed, etc). But if you spoof the outer SNI, that provides opportunities for easy detection that something suspicious is going on. So having no outer SNI might really be the best option for some people in this situation.

Hmm, well, assuming you can "just get a new IP" is quite an assumption, as most people don't have control over this, and if they do, it's not cheap. (Residential IPs = almost never; cloud servers = you can destroy and create instances, but this costs; having an IP range allocated to you = usually even more expensive).

Even if the outer SNI is spoofed or missing, ECH is advertised in the handshake, so snoops can always tell that something "suspicious" is going on... (And I want to emphasize that use of privacy features does not imply suspicious activity or content -- for the same reason we must avoid selective encryption, i.e. encrypting only sensitive data -- all connections are suspicious, all content is sensitive. That's the reality.)

@Helanzy

Interesting; so, you're suggesting that some residential ISPs block all TLS handshakes with a non-empty SNI? That seems odd, since if they want to block all HTTP, they should be blocking based on ALPN and/or port, not SNI. Are you sure that's correct?

> Is there any information in ECH's ClientHello to makes it different with existing TLS connections and thus can be detected?

The `encrypted_client_hello` extension (65037) is advertised.

---

Thank you, everyone, for the discussion. I think this is really good for development, and helpful for my understanding, since I don't have your perspectives on things.@shawnz: > Even if the outer SNI is spoofed or missing, ECH is advertised in the handshake, so snoops can always tell that something "suspicious" is going on...

Sure, but again that's what GREASE is for. Everyone should have GREASEd ECH in the handshake even if they're not using it, to mitigate that possibility@Helanzy: > Interesting; so, you're suggesting that some residential ISPs block all TLS handshakes with a non-empty SNI?

Direct TLS handshaking is not blocked. In fact occasional access to the web page is also tolerant, only that if hosting a long-lived web service and ISP detects it at a time, residential IP may get degraded accessibility, e.g. 1) block the port; 2) convert residential public IP to private NAT IP; 3) contact the owner for a fine or stop network service. SNI here is a proof for ISP that a web service exists.

> That seems odd, since if they want to block all HTTP, they should be blocking based on ALPN and/or port, not SNI.

Port 80/443 has already been banned and no connection can be made using this port. ALPN is indeed another way to detect that it's HTTP connection, but SNI helps ISP to do a replay to see if there's really a web service.

Besides, in [Cloudflare's blog](https://blog.cloudflare.com/encrypted-client-hello/#the-ins-and-outs-of-ech), ALPN is encrypted into ClientHelloInner in ECH, and ClientHelloOuter need not to specify it:

> It's not necessary that the client specify an ALPN list in the ClientHelloOuter, nor any other extension used to guide post-handshake behavior. All of these parameters are encapsulated by the encrypted ClientHelloInner.

Therefore, outer SNI might be the only issue for a non-HTTPs TLS handshake in this scenario.

> The encrypted_client_hello extension (65037) is advertised.

Okay, then ECH may not be a permanently safe way for such scenario. But given that `ClientHelloOuter` is designed to look like normal TLS connection to live with existing network ossification, making ECH without outer SNI looks like non-HTTPs TLS connection also makes it fit into existing network ossification in the long-term, since it does not influence ISP's basic functionality, which otherwise would make them aware of ECH and prompt them to deal with it.@christaikobo: > Sorry if I seem dismissive -- I want to be clear that I value your feedback and experience. I'm trying to _understand_ it, and for me, that understanding comes by way of scrutiny. It's how I challenge my current view of the topic. So please forgive me if I break it down, I really am just trying to make sure I am on the same page, since I don't believe there's much published literature on this topic yet. (I haven't found it if there is.) So I feel like we're forging ahead here.

Sorry I was a bit impatient as I was a bit frustrated, not being able to get my perspective across.

> > I have said earlier that for small organization and individual, hiding private SNI with public SNI is pretty much useless, because you are revealing an SNI anyway and if you only have one domain, it doesn't really matter if you are revealing "a.domain.com" or "b.domain.com".
> 
> If the outer SNI is not that of the services in the anonymity set, then what is the harm in it?

The restriction that residential user shouldn't host anything the other user mentioned applies as well, SNI is an indication of the existence of HTTPS server. I didn't want to talk about it because it is quite niche.

But more importantly, the thing about privacy is sometimes not always about immediate and tangible harm, which is unlike security. One of the major ISP in my country which has over a billion users used offer this service, in which you could print out all the website you have visited this billing period, that they logged via SNI. It's thousands of entries if not 10s of thousands of entries. And I don't like that at all. I am going to make an analogy: what harm does it have if some random person sees me naked? But I certainly don't want to be seen naked, even if no harm comes my way. 

> Also, can you help me understand why "if you only have one domain" is a relevant constraint? I don't think there's any reason you have to only have 1 domain. Do you mean _registered domain_?Even if you do only have one of those, maybe for cost reasons, you could serve the actual service at `service.example.com` but have the public name be `example.com` -- something generic. Or maybe `ech.example.com`. What does that reveal that can't already be inferred from casual inspection (e.g. IP address etc)?

The thing is a lot of these automated system will never do "casual inspection", they only do automated logging and/or blocking based off certain predefined rules. And SNI is a VERY vital part of these automated systems. They don't have the resources to "casually inspect" every user. Nor would they want to deploy such a unreliable method which could cause massive side effects on a large scale, which I explained in the last comment. 

About "if I only have one domain", if I have 10 domains, and I use `ech.domain-a.com`, I can at least hide the other 9 domains. Again, automated systems will not target me individually and "casually inspect" me, that is just not feasible. @mholt: > Sure, but again that's what GREASE is for. Everyone should have GREASEd ECH in the handshake even if they're not using it, to mitigate that possibility

Good point, but I also don't think I will reasonably expect that all major clients will implement GREASE... maybe, and it'd be cool if basically every TLS handshake included it. But I dunno, we'll see.

> But more importantly, the thing about privacy is sometimes not always about immediate and tangible harm, which is unlike security. One of the major ISP in my country which has over a billion users used offer this service, in which you could print out all the website you have visited this billing period, that they logged via SNI. It's thousands of entries if not 10s of thousands of entries. And I don't like that at all. I am going to make an analogy: what harm does it have if some random person sees me naked? But I certainly don't want to be seen naked, even if no harm comes my way.

And to clarify, I totally agree: there is a reason we wear clothes... nobody is disagreeing with that. We're just trying to figure out whether it's more private to wear a white shirt than a blue shirt or a graphic tee, etc.

> The thing is a lot of these automated system will never do "casual inspection", they only do automated logging and/or blocking based off certain predefined rules. And SNI is a VERY vital part of these automated systems. 

Right, but this doesn't explain why no SNI is better than an arbitrary non-empty one.

---

I feel like the argument for an empty outer SNI is still beating around the bush. _What's the actual privacy benefit that is so substantial it is worth making client connections brittle?_ So far, the claim remains unsubstantiated. What am I missing?@Gunni: Having a default of outer SNI enabled with a domain already managed by caddy makes sense, it's the most compatible way.

I just think having the _ability_ to have no outer SNI makes sense, even if it isn't the default. Just like being able to configure what TLS version are allowed, but TLS 1.2 (as configured by caddy by default) vs TLS 1.3 have no substantial privacy benefits, right?@mholt: @Gunni Gotcha -- although, to nit-pick, I disagree with:

>  Just like being able to configure what TLS version are allowed, but TLS 1.2 (as configured by caddy by default) vs TLS 1.3 have no substantial privacy benefits, right?

TLS 1.3 _does_ add provable privacy benefits (post-quantum cryptography, for example), and at least at the time, TLS 1.2 was still required for a vast majority of clients to even be compatible.

> I just think having the ability to have no outer SNI makes sense, even if it isn't the default.

To re-iterate what I said above, I hesitate to add this option without provable benefits (I added to my own quote in bold):

> I'm opposed to just doing this without a really compelling reason is because I'm worried about introducing security/privacy theater into the application. If people think they have more privacy without a public name, and that's not provably true, then that's harmful because it makes the protocol and the whole ecosystem more brittle, **[not to mention making people believe they have more privacy when they actually may not]**. Clients will fail to connect entirely -- or worse, resort to insecure methods of connecting as a last resort -- without a public SNI.

It is provably harmful to add this without substantial reason. What hasn't been proven yet is that the reason to add it is, in fact, a substantial privacy benefit.@Helanzy: > Right, but this doesn't explain why no SNI is better than an arbitrary non-empty one.

Per my understanding, for `ClientHelloOuter` to play the role to securely provide updated public key on decrypt failure, an encrypted TLS connection should be established using outer SNI. So that means a valid certificate shall be provided for outer SNI, which means we shall *own* the provided outer SNI domain, right?

Then, we cannot say it's an `arbitrary non-empty` domain. For home users, they may have to use their only domain `ech.mydomain` or `whatever.mydomain` as outer SNI. Then, the `whois` information, or regulated registrant for `mydomain`, or delegated subdomain `sub.mydomain` to regulated DNS provider, or personal page hosted at etld+1 `mydomain` may be enough to reveal their identity.

> I feel like the argument for an empty outer SNI is still beating around the bush. What's the actual privacy benefit that is so substantial it is worth making client connections brittle? So far, the claim remains unsubstantiated. What am I missing?

Since I didn't see quote regarding my comment, I assume this is a response to my comment. I am seeking ECH as an opportunity to build a TLS connection with HTTPS information private: 
1. Even ISP found ECH enabled in advertised extensions, they cannot tell whether it's HTTPS or not.
2. ISP cannot do a replay to see if there's a web service hosted on specific port.
3. ISP cannot trace domain owner's information using outer SNI.

ECH with outer SNI (with valid certificate) helps with 1 (ALPN is private now) and 2 (accessing `ech.mydomain` directly will be rejected), but 3 is not trace-proof (especially regarding owning the domain to provide a certificate). With empty outer SNI, 3 is ensured, while 1 is strengthened as there have been reports in my country's ISP using SNI as proof for existence of web service. On the contrary there are more ways to mitigate brittle client connection, e.g. using a private VPN channel to advertise public key in time.

@christaikobo: > > I'm opposed to just doing this without a really compelling reason is because I'm worried about introducing security/privacy theater into the application. If people think they have more privacy without a public name, and that's not provably true, then that's harmful because it makes the protocol and the whole ecosystem more brittle, **[not to mention making people believe they have more privacy when they actually may not]**. Clients will fail to connect entirely -- or worse, resort to insecure methods of connecting as a last resort -- without a public SNI.

How about putting the option in an addon? Anyone who uses default binary or docker won't have the option.

And anyone who actively compile the addon into caddy should be aware what they are doing. @mholt: > Per my understanding, for ClientHelloOuter to play the role to securely provide updated public key on decrypt failure, an encrypted TLS connection should be established using outer SNI. So that means a valid certificate shall be provided for outer SNI, which means we shall own the provided outer SNI domain, right?

Specifically, the server needs to be authoritative for the outer domain, yes.

But that should be no surprise. For example, of course a Cloudflare IP is going to own `cloudflare-ech.com`. Or you might own `alsdkflksjdflksjdf.com` at your residential IP. 🤷‍

> Then, we cannot say it's an arbitrary non-empty domain.

Sure, but what I meant is that it doesn't have to be related to the services being hidden. It can be anything, even a random ASCII string. But yes you're right it can't just be any domain name. It can be random, though.

>  For home users, they may have to use their only domain ech.mydomain or whatever.mydomain as outer SNI.

Huh, I was not aware of this restriction. Can you tell me more?

> Then, the whois information, or regulated registrant for mydomain, or delegated subdomain sub.mydomain to regulated DNS provider, or personal page hosted at etld+1 mydomain may be enough to reveal their identity.

So this is where I would say... don't do that.

(Also, is WHOIS information still publicly available? I wasn't aware it was still published. Just trying a few domains right now and I just get hidden/privacy masks in the output.)

But even so, that all misses the point.

ECH is not meant to hide associations with IP addresses. In fact, associations can be drawn even without domain names. Again, it is an orthogonal concern.

> ECH with outer SNI (with valid certificate) helps with 1 (ALPN is private now) and 2 (accessing ech.mydomain directly will be rejected), but 3 is not trace-proof (especially regarding owning the domain to provide a certificate). With empty outer SNI, 3 is ensured, while 1 is strengthened as there have been reports in my country's ISP using SNI as proof for existence of web service. On the contrary there are more ways to mitigate brittle client connection, e.g. using a private VPN channel to advertise public key in time.

Thanks for elaborating. However I don't think I see things the same way:

> they cannot tell whether it's HTTPS or not.

I _believe_ that should be true, however, be aware that most ISPs / powers-that-be just use port, since the vast majority of HTTPS traffic is on port 443.

> ISP cannot do a replay to see if there's a web service hosted on specific port.

Can you help me understand this better? The spec doesn't mention the word "replay" at _all_ so I would be surprised if this was a design goal.

> ISP cannot trace domain owner's information using outer SNI.

ISP does not need outer SNI to do this. :) IP address is even more informative.

> How about putting the option in an addon?

_Maybe_, but enabling support for a module here would be akin to advocating the capability, I feel like.@Helanzy: > But that should be no surprise. For example, of course a Cloudflare IP is going to own cloudflare-ech.com. Or you might own alsdkflksjdflksjdf.com at your residential IP. 🤷‍
>  It can be anything, even a random ASCII string. But yes you're right it can't just be any domain name. It can be random, though.
> > For home users, they may have to use their only domain `ech.mydomain` or `whatever.mydomain` as outer SNI.
> 
> Huh, I was not aware of this restriction. Can you tell me more?

The `it can be random domain` statement here is what I don't understand all along. The whole discussion is based on the statement `Client will establish TLS connection using outer SNI if failed to decrypt ClientHelloInner`, as stated in [Cloudflare's Blog](https://blog.cloudflare.com/encrypted-client-hello/#the-ins-and-outs-of-ech):

> At a minimum, both ClientHellos must contain the handshake parameters that are required for a server-authenticated key-exchange. In particular, while the ClientHelloInner contains the real SNI, the ClientHelloOuter also contains an SNI value, which the client **expects to verify** in case of ECH decryption failure (i.e., the client-facing server). If the connection is **established using the ClientHelloOuter**, then the client is expected to immediately abort the connection and retry the handshake with the public key provided by the server.

Then, from my understanding we need a certificate signed by acknowledged CAs for outer SNI, to make the handshake success. If we provide a self-signed certificate for random ASCII string, won't the browser just reject to establish the connection? 

For Cloudflare, they can prove their authority of `ech-cloudflare.com` and terminate the TLS connection for us, but for individuals with only single domain `mydomain`, isn't `ech.mydomain` or `whatever.mydomain` the only choice to get a widely-accepted certificate from providers like Let's Encrypt? 

In this way, outer SNI `ech.mydomain` is co-related with hidden service `service.mydomain` according to etld `mydomain`, which is not random like `alsdkflksjdflksjdf.com` (outer) and `service.mydomain` (inner).

> I believe that should be true, however, be aware that most ISPs / powers-that-be just use port, since the vast majority of HTTPS traffic is on port 443.

Since my ISP already blocked 80/443/8080/8443 port unconditionally, users have to use different port at high range like 12345 at the beginning. There are widely reported cases of my ISP that they detected web service in ports like 5000 (e.g. NAS's admin page) and contacted the user for regulation, so the port-443-only assumption is not relevant in my scenario.

> Can you help me understand this better? The spec doesn't mention the word "replay" at all so I would be surprised if this was a design goal.

The "replay" detection of web service can work in this way:
1. ISP sniffed out SNI `service.mydomain:port` from HTTPS handshake;
2. ISP visits `service.mydomain:port` (path is encrypted in HTTPS, so can only view index page).
3. A welcome page is shown -> web service found.

For HTTP, everything including path and response HTML can be detected. For HTTPS, only domain can be detected from SNI and certificate CN in handshake. For HTTPS with ECH (with outer SNI), only `ech.mydomain` can be detected, and visiting this domain will be rejected. If HTTPS with ECH (without outer SNI), then nothing can be found (looks like TLS on top of non-HTTPS applications).

> ISP does not need outer SNI to do this. :) IP address is even more informative.

You're right, so the point of ECH here is to completely hide the entrance to a web service (only `ech.mydomain` can be found), or in my expectation to completely behaves like non-HTTPS traffic.

--- 

Urrr, I found something contradictory in my expectation: If a certificate from recognized CA is always required, then how can we provide such a recognized certificate without outer SNI / with empty outer SNI? In [RFC](https://www.ietf.org/archive/id/draft-ietf-tls-esni-22.html#name-authenticating-for-the-publ) it seems authentication on outer SNI is required for client:

> [6.1.7. ](https://www.ietf.org/archive/id/draft-ietf-tls-esni-22.html#section-6.1.7)[Authenticating for the Public Name](https://www.ietf.org/archive/id/draft-ietf-tls-esni-22.html#name-authenticating-for-the-publ)
When the server rejects ECH, it continues with the handshake using the plaintext "server_name" extension instead (see [Section 7](https://www.ietf.org/archive/id/draft-ietf-tls-esni-22.html#server-behavior)). Clients that offer ECH then authenticate the connection with the public name, as follows:
>
> The client MUST verify that the certificate is valid for ECHConfig.contents.public_name. If invalid, it MUST abort the connection with the appropriate alert.
>
> If the server requests a client certificate, the client MUST respond with an empty Certificate message, denoting no client certificate.
>
> In verifying the client-facing server certificate, the client MUST interpret the public name as a DNS-based reference identity. Clients that incorporate DNS names and IP addresses into the same syntax (e.g. [[RFC3986](https://www.ietf.org/archive/id/draft-ietf-tls-esni-22.html#RFC3986)], [Section 7.4](https://rfc-editor.org/rfc/rfc3986#section-7.4) and [[WHATWG-IPV4](https://www.ietf.org/archive/id/draft-ietf-tls-esni-22.html#WHATWG-IPV4)]) MUST reject names that would be interpreted as IPv4 addresses. Clients that enforce this by checking ECHConfig.contents.public_name do not need to repeat the check at this layer.
> 
> Note that authenticating a connection for the public name does not authenticate it for the origin. The TLS implementation MUST NOT report such connections as successful to the application. It additionally MUST ignore all session tickets and session IDs presented by the server. These connections are only used to trigger retries, as described in [Section 6.1.6](https://www.ietf.org/archive/id/draft-ietf-tls-esni-22.html#rejected-ech). This may be implemented, for instance, by reporting a failed connection with a dedicated error code.







@christaikobo: > _Maybe_, but enabling support for a module here would be akin to advocating the capability, I feel like.

First of all, I see your concerns here. Caddy is a very popular tool and TLS is such a core component in its functionality that any change to it will have massive implications to the whole HTTP ecosystem. 

On the other hand, you guys must have statistical metrics about the percentage of users who just use your docker image or download precompile binary from official website/github, as opposed to the percentage of users who use online xcaddy to build custom caddy with addons. 

(Granted not all users use online build tool, but not all users download official docker image or precompiled binary either, so that kinda evens out)

And I would guess the ratio is quite extreme. My personal thought on this is if a very small percentage of users are willing to go out of their way to experiment on this with awareness of it being potentially breaking, the influence on the whole ecosystem should be limited. What is often harmful to the ecosystem is usually wrong defaults, which is not the case here. @mholt: @Helanzy 

> The "it can be random domain" statement here is what I don't understand all along. ...  from my understanding we need a certificate signed by acknowledged CAs for outer SNI, to make the handshake success. If we provide a self-signed certificate for random ASCII string, won't the browser just reject to establish the connection?

Yes, so to clarify, it needs to be able to get a valid certificate for the retry mechanism to succeed -- you _may_ use any random domain you own, I don't see why the contents of the domain  are significant? For example, you quoted me saying: "Or you might own `alsdkflksjdflksjdf.com`" -- that is a perfectly fine domain you can use for your outer SNI if you want to. It doesn't have to be, like, `helanzys-services.com`. The domain can be a totally random string.

> but for individuals with only single domain mydomain

And again, I'm stuck on why is this a requirement/restriction? Even if it is, what's the matter with that? Individual services are still masked behind a generic, not-service-specific domain.

> The "replay" detection of web service can work in this way:
>
> 1.   ISP sniffed out SNI service.mydomain:port from HTTPS handshake;
> 2.   ISP visits service.mydomain:port (path is encrypted in HTTPS, so can only view index page).
> 3.   A welcome page is shown -> web service found.

What? I follow step 1... and step 2... but step 3? The point of ECH isn't to hide web services. They are published publicly! The point of ECH is to make it difficult for wiretappers to know which service you are accessing.

@christaikobo Actually, we have no such information. We tried to count that once upon a time, but everyone lost their minds so we stopped.

---

Anyway, at this point, the claim that "ECH without an outer/public name is more private" has not been substantiated/proven, so I will not be implementing that at this time, to prevent potential harm to users and the ecosystem.

(Note I said, "at this time" -- if the claim _is_ proven or demonstrated in a real world scenario, then we can revisit this, and see if it is worth the trade-off of making the system more brittle.)@christaikobo: > Anyway, at this point, the claim that "ECH without an outer/public name is more private" has not been substantiated/proven, so I will not be implementing that at this time, to prevent potential harm to users and the ecosystem.
> 
> (Note I said, "at this time" -- if the claim _is_ proven or demonstrated in a real world scenario, then we can revisit this, and see if it is worth the trade-off of making the system more brittle.)

I fully understand and respect the decision from the dev. Love all the work you guys have been doing. Although a consensus is not reached this time, caddy team's swift response to industry evolution and open attitude towards user input are much appreciated. @gstrauss: @mholt wrote:
> I feel like the argument for an empty outer SNI is still beating around the bush. What's the actual privacy benefit that is so substantial it is worth making client connections brittle? So far, the claim remains unsubstantiated. What am I missing?

Perhaps those specifying "no outer SNI" are spitballing a "solution" without fully describing the problem they are trying to solve with that "solution"?

I understood only one attempt to explain "no outer SNI": I think @Helanzy is suggesting that a server configured for ECH which accepts TLS ClientHello without the TLS SNI extension, but with the TLS ECH extension *might not be* detected and blocked by an ISP as a web service.  If that is what @Helanzy is suggesting, then I would like to see a documented example.  That would be suggesting using TLS ECH to work around "all ISP-detected web services getting blocked" -- until the ISP wisened up -- even though that is not directly a privacy benefit of TLS ECH.

---

For others, you can have a wildcard cert for *.my-generic-domain.com as your public server name.  Use random words prepended to .my-generic-domain.com and you can have lots and lots of different public SNI names coming in as requests, enlarging your anonymity set to lots of different public SNI names (though with shared domain suffix).  The inner SNI can be anything else you like.

You can have a self-signed cert, or certs signed with a private CA (or private corporate CA), with lots of different SAN names, including spoofs of names of popular sites.  If the CA signature does not validate or is not trusted by a given client, then the client should not trust the certificate, but is that a concern for you for your ECH public name?  It could be a misconfiguration.  It could be corporate malware producing spoofed certificates for all sites so that the corporate malware could proxy (and view) all HTTPS traffic for the specific corporate clients which trusts the corporate CA.  Other than the situation where your ISP is state-owned and requires an explicit list of state-sanctioned CAs, most other ISPs are not able to enumerate all possible CAs.

---

If you want to have "secret" sites reachable *only* via ECH, and you do not want to expose retry configs, then that might provide an incremental level of privacy.

I am a lighttpd developer.  lighttpd implements TLS ECH with the ability to (attempt) to hide hosts configured in lighttpd.conf to be explicitly accessed via ECH-only.  I implemented the feature of ECH-only hosts in lighttpd as an experiment so that others could try it out and help substantiate whether or not it is measurably useful.

*IF* the ECH public key is distributed out-of-band instead of via public DNS, then an ECH-only service accessed could be "more private" than a typical service accessible via ECH and non-ECH connections.  State-level actors with broad surveillance capabilities can still detect that you are accessing a "secret" service located at a given IP, but might not know specifically which service, and won't not be able to spider the "secret" service without the public key.  (Requiring stronger authentication, which *is not* HTTP Basic Auth, to access the "secret" site is also strongly recommended.)@Helanzy: > If that is what @Helanzy is suggesting, then I would like to see a documented example.

@gstrauss There are some discussion in forums [[1]](https://www.v2ex.com/t/608821?p=3) [[2]](https://fast.v2ex.com/t/610260), and a blogged example that I'd like to specifically analyze: [Blog](https://blog.tangwudi.com/technology/homedatacenter12198) ([Translation](https://blog-tangwudi-com.translate.goog/technology/homedatacenter12198/?_x_tr_sl=zh-CN&_x_tr_tl=en&_x_tr_hl=zh-CN&_x_tr_pto=wapp)):

The author's network got suspended by their ISP for hosting following **unlicensed** domain/port detected in ISP's feedback:
1. wow.wangwudi.com
2. https://wow.wangwudi.com:8443

Where:
1. `wow.wangwudi.com` is domain of a self-hosted World of Warcraft server resolved to author's residential IP, hosting no HTTP(s) service.
2. `:8443` is a remote admin page of a router.

Therefore, `wow.wangwudi.com:8443` is an unreasonable combination, which is reported to derive from scanning the ports on author's residential IP. `wow.wangwudi.com` is likely to be leaked from DNS requests generated in DDNS according to investigation.

Here we need to bring out the concept of `ICP License`. For domains to legitimately host services in China, they must be granted with ICP License. However ICP licenses for domains resolved to residential IPs are practically unobtainable.

So in summary, when a domain is detected to associate with a residential IP (either via SNI or DNS), and with port scanning a web service is found, then ISP would generally suspend the network for the user. 

> Interesting; so, you're suggesting that some residential ISPs block all TLS handshakes with a non-empty SNI? That seems odd, since if they want to block all HTTP, they should be blocking based on ALPN and/or port, not SNI. Are you sure that's correct?

@mholt Since ISP needs to check whether a domain is holding a license, SNI instead of ALPN is preferred.

> Right, but this doesn't explain why no SNI is better than an arbitrary non-empty one.
>
> Or you might own alsdkflksjdflksjdf.com at your residential IP. 🤷‍ 
>
> > but for individuals with only single domain mydomain
>
> I'm stuck on why is this a requirement/restriction?

Burden to maintain one more domain aside, neither `ech.mydomain` nor `alsdkflksjdflksjdf.com` is a licensed domain, so once detected they plays no difference - you can reject any connection on these domains to not provide service, though. For ECH without outer SNI, then ISP cannot find any domain to check whether it's licensed or not.

---

A concern for it lies on implementation side. Since we cannot provide a valid license for empty SNI, server may send back `HelloRetryRequest` or reject it. But for client side, what will happen with empty SNI is not guaranteed. Clients are more likely to not omit the SNI extension in handshake, or just reject with error.@gstrauss: @Helanzy thank you for the details.  While I encourage your research, the use case you presented is unrelated to "implementing TLS ECH according to the draft specification".

The present answer to @mholt's question: "What's the actual privacy benefit that is so substantial it is worth making client connections brittle?" is that there is no *direct* privacy benefit which has, as yet, been described.@mholt: @christaikobo 

> I fully understand and respect the decision from the dev. Love all the work you guys have been doing. Although a consensus is not reached this time, caddy team's swift response to industry evolution and open attitude towards user input are much appreciated.

Thanks for holding a respectful conversation, even though I'm sure it must be frustrating. :)

@Helanzy 

Thanks for the detailed use case. That's very helpful, as I'm otherwise unaware how that works in China.

@gstrauss

> I am a lighttpd developer.

First, thank you for chiming in! Feedback from another web server developer implementing ECH is as about as valuable as it gets for me right now. It's good to know I'm not alone. 😅 Thanks for bringing the focus onto the question at hand and summing things up for me with your knowledge.

So, at this point, my understanding is that:

- It can be desirable to keep which service(s) a host is operating private from parties who are able to observe all incoming and outgoing network activity as it happens (this includes, and is not limited to: origin and destination IPs, number of bytes, protocols used, any unencrypted data, and patterns of encrypted data).

- Having no outer SNI is not necessarily more private with respect to ECH, but if the desire is as stated in the last bullet point, omitting all plaintext server names would be beneficial.

- However, as a certificate is still needed for the inner SNI, a certificate must be procured by the server. This requires DNS records pointing to the machine, which eliminates pure anonymity, OR a DNS challenge record created to prove control over a domain. These records are (generally? always?) created in the clear and at specific times that can be observed on the network.

- Even if A and AAAA records don't exist for the inner domain, an observer of the network can correlate certificate issuance times with certificate transparency (CT) logs to ascertain which domain is being served at the observed host (even if timestamps aren't precise, it vastly narrows the search space).

- Thus, keeping the host's service(s) truly anonymous would require the use of a private/internal CA, that does not publish to CT logs, and whose root would have to be trusted/installed by any/all clients who wish to connect to such services. In other words, as the operator of the private service, you would have to: 1) distribute ECHConfigs in such a way that no adversary could discover, 2) create no public DNS records, 3) obtain no publicly-trusted certificate, 4) operate your own internal/private CA, and 5) ensure the CA's root is trusted by all clients of your service.

Clearly, this falls well beyond the scope of ECH, although ECH would be a crucial component of such a setup. This is the ONLY case I can think of where omitting an outer SNI would provide the privacy benefits being requested (I think); it would essentially eliminate all public traces of the service(s).

(Let me know if I have misunderstood or misrepresented anything!)@Gunni: You may be forgetting Wild Card DNS records with wild card certificates can completely mask which domain names are in use.@gstrauss: @mholt, I think you have expressed well that TLS ECH can be *part* of solutions making the web more private, but that there are many, many more details.  Anyone looking at TLS ECH as "the solution" to a problem probably ought to first focus on VPNs, proxies, and other anonymity services before considering TLS ECH.

The Caddy feature for auto-procurement of certificates may leave fingerprints which make hiding the existence of TLS ECH-only hosts more difficult, ... if ECH-only services are even something that can be reasonably achieved (TBD).

---

Regarding TLS ECH implementation, I have not looked into the recently added Go support.  Still, I want to highlight a potential privacy knob.  There is a random byte in the ECHConfig which can help optimize selection of the key on the server used to decrypt the TLS ECH inner, and this feature is desirable to large CDNs.  The RFC draft recognizes that this byte is a possible tracking vector.  The defo.ie patches for OpenSSL include an option for ECHTrialDecrypt to enable trial decryption using a list of keys on the server.  Whether or not to use the ECHConfig random byte -- which should be changed when ECHConfig keys are rotated -- is something to consider, as is whether or not to avoid using random byte (by always pinning to same value) and to force trial decryption to find matching keys.

lighttpd TLS ECH doc includes some notes on using OpenSSL and BoringSSL clients to generate ECHConfig:
https://wiki.lighttpd.net/TLS_ECH
OpenSSL and BoringSSL currently have slight differences in the precise format (wrapping) the generated ECHConfig.
The doc also includes an example of how to test TLS ECH using the cURL client and using private ECHConfig which are not published in DNS.

ECHConfig key rotation should be frequent (at least multiple times per day) and that adds work for servers, including Caddy, to retrieve and rotate keys, including possibly supporting recently-rotated keys for a short period of overlap.

Additional reference: https://github.com/defo-project 
Stephen Farrell (@sftcd) has been one of the primary implementers developing and submitting patches with TLS ECH support to various projects.  He is very approachable if you have questions about TLS ECH.@sftcd: Happy to help out with ECH here if I can - just ping if there's something I can help with.

IIRC the next golang release was supposed to include the ECH server code, so it should be there now or soon, and our experience is that it's not that hard to ECH-enable a server once you figure out where to put the few new ECH  API calls. (But still happy to help with that if help's useful.)
@mholt: @Gunni 

> You may be forgetting Wild Card DNS records with wild card certificates can completely mask which domain names are in use.

I thought about those, but they would still be subject to the same issuance-time correlation side-channel of any public certificates.

@gstrauss 

Thanks, really great information there!

>  There is a random byte in the ECHConfig which can help optimize selection of the key on the server used to decrypt the TLS ECH inner, and this feature is desirable to large CDNs. The RFC draft recognizes that this byte is a possible tracking vector.

Are you referring to the `config_id`?

> ECHConfig key rotation should be frequent (at least multiple times per day) and that adds work for servers, including Caddy, to retrieve and rotate keys, including possibly supporting recently-rotated keys for a short period of overlap.

That's the plan -- although, I don't know about multiple times _per day_ 😅 I know the spec recommends "frequently" rotating keys, but we can probably manage either way.

@sftcd Nice to meet you, thanks for popping in!

> IIRC the next golang release was supposed to include the ECH server code, so it should be there now or soon, and our experience is that it's not that hard to ECH-enable a server once you figure out where to put the few new ECH API calls. 

Indeed, Go 1.24 supports ECH on the server-side, and its API is very simple. Basically just set [this field](https://pkg.go.dev/crypto/tls#Config.EncryptedClientHelloKeys) in a `tls.Config`.

I'm currently designing the config API for enabling ECH ub Caddy, but then I'll surely have questions as I come to more ECH logistics, such as key rotation. I really appreciate your willingness to chime in :+1:

I'll also hopefully have a WIP PR to share in the next week or two.@gstrauss: > > There is a random byte in the ECHConfig which can help optimize selection of the key on the server used to decrypt the TLS ECH inner, and this feature is desirable to large CDNs. The RFC draft recognizes that this byte is a possible tracking vector.
> 
> Are you referring to the `config_id`?

Yes.@Gunni: @mholt

> I thought about those, but they would still be subject to the same issuance-time correlation side-channel of any public certificates.

Yes but you might f.ex get a wildcard cert for example.com, and then only allow specific subdomains to connect, then `close` the connection for others.@sftcd: 

On 20/02/2025 18:29, 古大羊 wrote:
> In my view, the ECH RFC standard is one aspect, while client-side
> support (such as in browsers) is another. Therefore, the claim that
> ECH can hide external domain names or use third-party domain names
> that do not belong to it is unrealistic. Server-side implementation
> must not only adhere to the ECH specification but also consider the
> actual compatibility of clients (browsers); otherwise, it is merely
> empty talk.

Not sure I'm following the above quite, but FWIW, browsers
(FF and chromium-based) have supported ECH for quite a
while, and cloudflare turned it on in their services (again)
last October. So in terms of protocol level interop, things
work just fine. There are some deployment wrinkles still
but I'd expect those to improve over time. We did an
interop report recently [1] calling out some of those.
(BTW [1] didn't include a golang server, but we'll likely
add one when we revise that in future.)

    [1] https://test.defo.ie/ech-interop-report.pdf

In terms of how much of a privacy benefit one gets from ECH,
yes that'll depend on which outer-sni (public_name) is
used, how that maps to a set of names usable as inner-sni
via that, how IP addresses are mapped to those names and
how identifiable DNS queries may be. My argument would be
that even if all of those involve unknowns, we're still
doing the right thing if we make it harder to abuse the
SNI value by encrypting that (via ECH). It's an incremental
improvement in privacy, but a worthwhile one that should
enable others in future.

Cheers,
S.
@mholt: Thanks @sftcd  -- great info!

I'm making progress on this, but recently hit a snag... the crypto/tls package expects us to hard-code the ECH keys into a TLS server config, rather than provide a callback function like it lets us do for certificates. That means the keys can't change during the TLS server lifetime, meaning they can't be rotated.

I hesitate to proceed with a design that doesn't support key rotation, because then if it ever is possible in the future, I may have to refactor a lot of code. (Or not; but it's uncertain.)

I might be totally doing something wrong or overlooking something obvious, but I opened an issue upstream just in case: https://github.com/golang/go/issues/71920@gstrauss: @mholt, I do not know enough of the Go language to comment on the bug report, but want to highlight the importance of having the ability to support multiple ECH keys for trial decryption.  Whether multiple keys are due to key rotation of the same host, or if multiple keys are due to multiple different hosts, sometimes trial decryption is necessary to decrypt the TLS ECH inner, and you will not be able to read SNI from the TLS ECH inner until *after* TLS ECH inner has been decrypted, which is *after* (potentially multiple different) decryption keys have been tried.  Does Go provide a mechanism to configure a list of multiple decryption keys?  Your bug report notes that Go does not provide server side support for dynamic reconfiguration of TLS ECH keys, and if what I am trying to describe is another limitation of the Go TLS ECH server-side implementation, it would be nice to provide that feedback so that the Go devs can suggest a solution to both issues.@mholt: @gstrauss Thanks -- Go does allow configuring multiple keys: https://pkg.go.dev/crypto/tls#Config.EncryptedClientHelloKeys

The 'EncryptedClientHelloKeys' field lets us specify multiple, which we will be doing in Caddy.@gstrauss: https://pkg.go.dev/crypto/tls#Config.EncryptedClientHelloKeys
```
	// If EncryptedClientHelloKeys is set, MinVersion, if set, must be
	// VersionTLS13.
```

Does that mean that servers on top of Go will require TLSv1.3 as minimum protocol -- and will not be able to support TLSv1.2 -- or does it mean that TLS ECH is available only with TLSv1.3, as TLS ECH is an extension which requires TLSv1.3?@sftcd: > I might be totally doing something wrong or overlooking something obvious, but I opened an issue upstream just in case: [golang/go#71920](https://github.com/golang/go/issues/71920)

The bug report looks valid to me (modulo my relative golang ignorance:-). In 'C'/OpenSSL you have the difference between the `SSL_CTX` context structure and the per-connection `SSL` structure so can handle ECH key rotation in the `SSL_CTX`, but I don't see a similar thing at https://pkg.go.dev/crypto/tls#Config.EncryptedClientHelloKeys  In any case, handling ECH key rotation similarly to TLS server private key/cert re-load seems like a good idea, esp given that cloudflare are rotating ECH keys hourly, its not unlikely others will emulate that.

As an aside, that https://pkg.go.dev/crypto/tls#Config thing looks like it might grow up to be like some of the internal OpenSSL structures (i.e. many too many fields) - avoiding that might be a good thing, were it possible:-)

And just for completeness - I very much doubt you'd want to, but haproxy has yet another way to handle key rotation for TLS server keys (and, in my PoC integration ECH keys) - you connect to a `stats` port and use `socat` to load in new private/public values. IIUC their logic in doing that is they don't want to depend on reading the disk any more once haproxy has started (so they continue in the face of some errors).@mholt: If anyone would like to try my WIP implementation, here it is: #6862 @mholt: ECH support has been merged and will soon be released with 2.10-beta.1. (We'll have many beta versions for this cycle as I continue to work through the backlog of PRs.)

It's more code than I expected (1500+ LoC), but I tested to make sure at least a few configurations worked properly. Everyone please give it a try!

Note that key rotation is deferred until after Go 1.25 is released, as it's not exactly feasible with Go 1.24.

If you encounter any bugs with it (you surely will), or want to start a new in-depth discussion, please open a new issue for those. Feel free to continue replying here with any smaller questions or comments you may have though.

@gstrauss 

> Does that mean that servers on top of Go will require TLSv1.3 as minimum protocol -- and will not be able to support TLSv1.2 -- or does it mean that TLS ECH is available only with TLSv1.3, as TLS ECH is an extension which requires TLSv1.3?

Hmm, I read this as TLS 1.3 must be the minimum in order to configure ECH at all, so that's what I've done. I haven't actually tried letting the minimum be TLS 1.2. I suspect that might be a broken config though, since you're asking Go to serve ECH with an incompatible TLS version.

Also, thanks for your valuable insights @sftcd -- I've learned a lot!@sftcd: good stuff - I might try do a few tests, but what's the timeframe for those to be most useful? (i.e. how long is a "cycle") @mholt: By cycle I just mean the 2.10 release in this case. No specific timeframe yet. 👍

Solution patch:
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 7f8e6d50..ef7eff94 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -18,22 +18,18 @@ jobs:
       # Default is true, cancels jobs for other platforms in the matrix if one fails
       fail-fast: false
       matrix:
-        os: 
+        os:
           - linux
           - mac
           - windows
-        go: 
-          - '1.23'
+        go:
           - '1.24'
 
         include:
         # Set the minimum Go patch version for the given Go minor
         # Usable via ${{ matrix.GO_SEMVER }}
-        - go: '1.23'
-          GO_SEMVER: '~1.23.6'
-
         - go: '1.24'
-          GO_SEMVER: '~1.24.0'
+          GO_SEMVER: '~1.24.1'
 
         # Set some variables per OS, usable via ${{ matrix.VAR }}
         # OS_LABEL: the VM label from GitHub Actions (see https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners/about-github-hosted-runners#standard-github-hosted-runners-for-public-repositories)
diff --git a/.github/workflows/cross-build.yml b/.github/workflows/cross-build.yml
index 231e4a6e..64ad77bd 100644
--- a/.github/workflows/cross-build.yml
+++ b/.github/workflows/cross-build.yml
@@ -15,7 +15,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        goos: 
+        goos:
           - 'aix'
           - 'linux'
           - 'solaris'
@@ -26,18 +26,14 @@ jobs:
           - 'windows'
           - 'darwin'
           - 'netbsd'
-        go: 
-          - '1.23'
+        go:
           - '1.24'
 
         include:
         # Set the minimum Go patch version for the given Go minor
         # Usable via ${{ matrix.GO_SEMVER }}
-        - go: '1.23'
-          GO_SEMVER: '~1.23.6'
-
         - go: '1.24'
-          GO_SEMVER: '~1.24.0'
+          GO_SEMVER: '~1.24.1'
 
     runs-on: ubuntu-latest
     continue-on-error: true
diff --git a/.github/workflows/lint.yml b/.github/workflows/lint.yml
index 3cfe893d..0766f78a 100644
--- a/.github/workflows/lint.yml
+++ b/.github/workflows/lint.yml
@@ -63,5 +63,5 @@ jobs:
       - name: govulncheck
         uses: golang/govulncheck-action@v1
         with:
-          go-version-input: '~1.24.0'
+          go-version-input: '~1.24.1'
           check-latest: true
diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml
index df42679b..483d1d70 100644
--- a/.github/workflows/release.yml
+++ b/.github/workflows/release.yml
@@ -19,7 +19,7 @@ jobs:
         # Set the minimum Go patch version for the given Go minor
         # Usable via ${{ matrix.GO_SEMVER }}
         - go: '1.24'
-          GO_SEMVER: '~1.24.0'
+          GO_SEMVER: '~1.24.1'
 
     runs-on: ${{ matrix.os }}
     # https://github.com/sigstore/cosign/issues/1258#issuecomment-1002251233
diff --git a/caddyconfig/httpcaddyfile/builtins.go b/caddyconfig/httpcaddyfile/builtins.go
index 45570d01..3fc08b2c 100644
--- a/caddyconfig/httpcaddyfile/builtins.go
+++ b/caddyconfig/httpcaddyfile/builtins.go
@@ -99,7 +99,7 @@ func parseBind(h Helper) ([]ConfigValue, error) {
 //	    ca                            <acme_ca_endpoint>
 //	    ca_root                       <pem_file>
 //	    key_type                      [ed25519|p256|p384|rsa2048|rsa4096]
-//	    dns                           <provider_name> [...]
+//	    dns                           [<provider_name> [...]]    (required, though, if DNS is not configured as global option)
 //	    propagation_delay             <duration>
 //	    propagation_timeout           <duration>
 //	    resolvers                     <dns_servers...>
@@ -312,10 +312,6 @@ func parseTLS(h Helper) ([]ConfigValue, error) {
 			certManagers = append(certManagers, certManager)
 
 		case "dns":
-			if !h.NextArg() {
-				return nil, h.ArgErr()
-			}
-			provName := h.Val()
 			if acmeIssuer == nil {
 				acmeIssuer = new(caddytls.ACMEIssuer)
 			}
@@ -325,12 +321,19 @@ func parseTLS(h Helper) ([]ConfigValue, error) {
 			if acmeIssuer.Challenges.DNS == nil {
 				acmeIssuer.Challenges.DNS = new(caddytls.DNSChallengeConfig)
 			}
-			modID := "dns.providers." + provName
-			unm, err := caddyfile.UnmarshalModule(h.Dispenser, modID)
-			if err != nil {
-				return nil, err
+			// DNS provider configuration optional, since it may be configured globally via the TLS app with global options
+			if h.NextArg() {
+				provName := h.Val()
+				modID := "dns.providers." + provName
+				unm, err := caddyfile.UnmarshalModule(h.Dispenser, modID)
+				if err != nil {
+					return nil, err
+				}
+				acmeIssuer.Challenges.DNS.ProviderRaw = caddyconfig.JSONModuleObject(unm, "name", provName, h.warnings)
+			} else if h.Option("dns") == nil {
+				// if DNS is omitted locally, it needs to be configured globally
+				return nil, h.ArgErr()
 			}
-			acmeIssuer.Challenges.DNS.ProviderRaw = caddyconfig.JSONModuleObject(unm, "name", provName, h.warnings)
 
 		case "resolvers":
 			args := h.RemainingArgs()
diff --git a/caddyconfig/httpcaddyfile/httptype.go b/caddyconfig/httpcaddyfile/httptype.go
index 37a6f6b2..69d88df0 100644
--- a/caddyconfig/httpcaddyfile/httptype.go
+++ b/caddyconfig/httpcaddyfile/httptype.go
@@ -1121,6 +1121,12 @@ func consolidateConnPolicies(cps caddytls.ConnectionPolicies) (caddytls.Connecti
 					return nil, fmt.Errorf("two policies with same match criteria have conflicting default SNI: %s vs. %s",
 						cps[i].DefaultSNI, cps[j].DefaultSNI)
 				}
+				if cps[i].FallbackSNI != "" &&
+					cps[j].FallbackSNI != "" &&
+					cps[i].FallbackSNI != cps[j].FallbackSNI {
+					return nil, fmt.Errorf("two policies with same match criteria have conflicting fallback SNI: %s vs. %s",
+						cps[i].FallbackSNI, cps[j].FallbackSNI)
+				}
 				if cps[i].ProtocolMin != "" &&
 					cps[j].ProtocolMin != "" &&
 					cps[i].ProtocolMin != cps[j].ProtocolMin {
@@ -1161,6 +1167,9 @@ func consolidateConnPolicies(cps caddytls.ConnectionPolicies) (caddytls.Connecti
 				if cps[i].DefaultSNI == "" && cps[j].DefaultSNI != "" {
 					cps[i].DefaultSNI = cps[j].DefaultSNI
 				}
+				if cps[i].FallbackSNI == "" && cps[j].FallbackSNI != "" {
+					cps[i].FallbackSNI = cps[j].FallbackSNI
+				}
 				if cps[i].ProtocolMin == "" && cps[j].ProtocolMin != "" {
 					cps[i].ProtocolMin = cps[j].ProtocolMin
 				}
diff --git a/caddyconfig/httpcaddyfile/options.go b/caddyconfig/httpcaddyfile/options.go
index d4a42462..52767687 100644
--- a/caddyconfig/httpcaddyfile/options.go
+++ b/caddyconfig/httpcaddyfile/options.go
@@ -19,6 +19,7 @@ import (
 	"strconv"
 
 	"github.com/caddyserver/certmagic"
+	"github.com/libdns/libdns"
 	"github.com/mholt/acmez/v3/acme"
 
 	"github.com/caddyserver/caddy/v2"
@@ -45,7 +46,7 @@ func init() {
 	RegisterGlobalOption("ocsp_interval", parseOptDuration)
 	RegisterGlobalOption("acme_ca", parseOptSingleString)
 	RegisterGlobalOption("acme_ca_root", parseOptSingleString)
-	RegisterGlobalOption("acme_dns", parseOptACMEDNS)
+	RegisterGlobalOption("acme_dns", parseOptDNS)
 	RegisterGlobalOption("acme_eab", parseOptACMEEAB)
 	RegisterGlobalOption("cert_issuer", parseOptCertIssuer)
 	RegisterGlobalOption("skip_install_trust", parseOptTrue)
@@ -62,6 +63,8 @@ func init() {
 	RegisterGlobalOption("log", parseLogOptions)
 	RegisterGlobalOption("preferred_chains", parseOptPreferredChains)
 	RegisterGlobalOption("persist_config", parseOptPersistConfig)
+	RegisterGlobalOption("dns", parseOptDNS)
+	RegisterGlobalOption("ech", parseOptECH)
 }
 
 func parseOptTrue(d *caddyfile.Dispenser, _ any) (any, error) { return true, nil }
@@ -238,25 +241,6 @@ func parseOptDuration(d *caddyfile.Dispenser, _ any) (any, error) {
 	return caddy.Duration(dur), nil
 }
 
-func parseOptACMEDNS(d *caddyfile.Dispenser, _ any) (any, error) {
-	if !d.Next() { // consume option name
-		return nil, d.ArgErr()
-	}
-	if !d.Next() { // get DNS module name
-		return nil, d.ArgErr()
-	}
-	modID := "dns.providers." + d.Val()
-	unm, err := caddyfile.UnmarshalModule(d, modID)
-	if err != nil {
-		return nil, err
-	}
-	prov, ok := unm.(certmagic.DNSProvider)
-	if !ok {
-		return nil, d.Errf("module %s (%T) is not a certmagic.DNSProvider", modID, unm)
-	}
-	return prov, nil
-}
-
 func parseOptACMEEAB(d *caddyfile.Dispenser, _ any) (any, error) {
 	eab := new(acme.EAB)
 	d.Next() // consume option name
@@ -570,3 +554,68 @@ func parseOptPreferredChains(d *caddyfile.Dispenser, _ any) (any, error) {
 	d.Next()
 	return caddytls.ParseCaddyfilePreferredChainsOptions(d)
 }
+
+func parseOptDNS(d *caddyfile.Dispenser, _ any) (any, error) {
+	d.Next() // consume option name
+
+	if !d.Next() { // get DNS module name
+		return nil, d.ArgErr()
+	}
+	modID := "dns.providers." + d.Val()
+	unm, err := caddyfile.UnmarshalModule(d, modID)
+	if err != nil {
+		return nil, err
+	}
+	switch unm.(type) {
+	case libdns.RecordGetter,
+		libdns.RecordSetter,
+		libdns.RecordAppender,
+		libdns.RecordDeleter:
+	default:
+		return nil, d.Errf("module %s (%T) is not a libdns provider", modID, unm)
+	}
+	return unm, nil
+}
+
+func parseOptECH(d *caddyfile.Dispenser, _ any) (any, error) {
+	d.Next() // consume option name
+
+	ech := new(caddytls.ECH)
+
+	publicNames := d.RemainingArgs()
+	for _, publicName := range publicNames {
+		ech.Configs = append(ech.Configs, caddytls.ECHConfiguration{
+			OuterSNI: publicName,
+		})
+	}
+	if len(ech.Configs) == 0 {
+		return nil, d.ArgErr()
+	}
+
+	for nesting := d.Nesting(); d.NextBlock(nesting); {
+		switch d.Val() {
+		case "dns":
+			if !d.Next() {
+				return nil, d.ArgErr()
+			}
+			providerName := d.Val()
+			modID := "dns.providers." + providerName
+			unm, err := caddyfile.UnmarshalModule(d, modID)
+			if err != nil {
+				return nil, err
+			}
+			ech.Publication = append(ech.Publication, &caddytls.ECHPublication{
+				Configs: publicNames,
+				PublishersRaw: caddy.ModuleMap{
+					"dns": caddyconfig.JSON(caddytls.ECHDNSPublisher{
+						ProviderRaw: caddyconfig.JSONModuleObject(unm, "name", providerName, nil),
+					}, nil),
+				},
+			})
+		default:
+			return nil, d.Errf("ech: unrecognized subdirective '%s'", d.Val())
+		}
+	}
+
+	return ech, nil
+}
diff --git a/caddyconfig/httpcaddyfile/tlsapp.go b/caddyconfig/httpcaddyfile/tlsapp.go
index 71b52492..adac1506 100644
--- a/caddyconfig/httpcaddyfile/tlsapp.go
+++ b/caddyconfig/httpcaddyfile/tlsapp.go
@@ -359,6 +359,30 @@ func (st ServerType) buildTLSApp(
 		tlsApp.Automation.OnDemand = onDemand
 	}
 
+	// set up "global" (to the TLS app) DNS provider config
+	if globalDNS, ok := options["dns"]; ok && globalDNS != nil {
+		tlsApp.DNSRaw = caddyconfig.JSONModuleObject(globalDNS, "name", globalDNS.(caddy.Module).CaddyModule().ID.Name(), nil)
+	}
+
+	// set up ECH from Caddyfile options
+	if ech, ok := options["ech"].(*caddytls.ECH); ok {
+		tlsApp.EncryptedClientHello = ech
+
+		// outer server names will need certificates, so make sure they're included
+		// in an automation policy for them that applies any global options
+		ap, err := newBaseAutomationPolicy(options, warnings, true)
+		if err != nil {
+			return nil, warnings, err
+		}
+		for _, cfg := range ech.Configs {
+			ap.SubjectsRaw = append(ap.SubjectsRaw, cfg.OuterSNI)
+		}
+		if tlsApp.Automation == nil {
+			tlsApp.Automation = new(caddytls.AutomationConfig)
+		}
+		tlsApp.Automation.Policies = append(tlsApp.Automation.Policies, ap)
+	}
+
 	// if the storage clean interval is a boolean, then it's "off" to disable cleaning
 	if sc, ok := options["storage_check"].(string); ok && sc == "off" {
 		tlsApp.DisableStorageCheck = true
@@ -553,7 +577,8 @@ func fillInGlobalACMEDefaults(issuer certmagic.Issuer, options map[string]any) e
 	if globalPreferredChains != nil && acmeIssuer.PreferredChains == nil {
 		acmeIssuer.PreferredChains = globalPreferredChains.(*caddytls.ChainPreference)
 	}
-	if globalHTTPPort != nil && (acmeIssuer.Challenges == nil || acmeIssuer.Challenges.HTTP == nil || acmeIssuer.Challenges.HTTP.AlternatePort == 0) {
+	// only configure alt HTTP and TLS-ALPN ports if the DNS challenge is not enabled (wouldn't hurt, but isn't necessary since the DNS challenge is exclusive of others)
+	if globalHTTPPort != nil && (acmeIssuer.Challenges == nil || acmeIssuer.Challenges.DNS == nil) && (acmeIssuer.Challenges == nil || acmeIssuer.Challenges.HTTP == nil || acmeIssuer.Challenges.HTTP.AlternatePort == 0) {
 		if acmeIssuer.Challenges == nil {
 			acmeIssuer.Challenges = new(caddytls.ChallengesConfig)
 		}
@@ -562,7 +587,7 @@ func fillInGlobalACMEDefaults(issuer certmagic.Issuer, options map[string]any) e
 		}
 		acmeIssuer.Challenges.HTTP.AlternatePort = globalHTTPPort.(int)
 	}
-	if globalHTTPSPort != nil && (acmeIssuer.Challenges == nil || acmeIssuer.Challenges.TLSALPN == nil || acmeIssuer.Challenges.TLSALPN.AlternatePort == 0) {
+	if globalHTTPSPort != nil && (acmeIssuer.Challenges == nil || acmeIssuer.Challenges.DNS == nil) && (acmeIssuer.Challenges == nil || acmeIssuer.Challenges.TLSALPN == nil || acmeIssuer.Challenges.TLSALPN.AlternatePort == 0) {
 		if acmeIssuer.Challenges == nil {
 			acmeIssuer.Challenges = new(caddytls.ChallengesConfig)
 		}
diff --git a/context.go b/context.go
index d4d7afac..94623df7 100644
--- a/context.go
+++ b/context.go
@@ -385,6 +385,17 @@ func (ctx Context) LoadModuleByID(id string, rawMsg json.RawMessage) (any, error
 		return nil, fmt.Errorf("module value cannot be null")
 	}
 
+	// if this is an app module, keep a reference to it,
+	// since submodules may need to reference it during
+	// provisioning (even though the parent app module
+	// may not be fully provisioned yet; this is the case
+	// with the tls app's automation policies, which may
+	// refer to the tls app to check if a global DNS
+	// module has been configured for DNS challenges)
+	if appModule, ok := val.(App); ok {
+		ctx.cfg.apps[id] = appModule
+	}
+
 	ctx.ancestry = append(ctx.ancestry, val)
 
 	if prov, ok := val.(Provisioner); ok {
@@ -471,7 +482,6 @@ func (ctx Context) App(name string) (any, error) {
 	if appRaw != nil {
 		ctx.cfg.AppsRaw[name] = nil // allow GC to deallocate
 	}
-	ctx.cfg.apps[name] = modVal.(App)
 	return modVal, nil
 }
 
diff --git a/go.mod b/go.mod
index 729109fd..f4914bff 100644
--- a/go.mod
+++ b/go.mod
@@ -8,8 +8,9 @@ require (
 	github.com/Masterminds/sprig/v3 v3.3.0
 	github.com/alecthomas/chroma/v2 v2.14.0
 	github.com/aryann/difflib v0.0.0-20210328193216-ff5ff6dc229b
-	github.com/caddyserver/certmagic v0.21.7
+	github.com/caddyserver/certmagic v0.21.8-0.20250220203412-a7894dd6992d
 	github.com/caddyserver/zerossl v0.1.3
+	github.com/cloudflare/circl v1.3.3
 	github.com/dustin/go-humanize v1.0.1
 	github.com/go-chi/chi/v5 v5.0.12
 	github.com/google/cel-go v0.21.0
@@ -36,11 +37,11 @@ require (
 	go.uber.org/automaxprocs v1.6.0
 	go.uber.org/zap v1.27.0
 	go.uber.org/zap/exp v0.3.0
-	golang.org/x/crypto v0.31.0
+	golang.org/x/crypto v0.33.0
 	golang.org/x/crypto/x509roots/fallback v0.0.0-20241104001025-71ed71b4faf9
 	golang.org/x/net v0.33.0
-	golang.org/x/sync v0.10.0
-	golang.org/x/term v0.27.0
+	golang.org/x/sync v0.11.0
+	golang.org/x/term v0.29.0
 	golang.org/x/time v0.7.0
 	gopkg.in/natefinch/lumberjack.v2 v2.2.1
 	gopkg.in/yaml.v3 v3.0.1
@@ -114,7 +115,7 @@ require (
 	github.com/jackc/pgservicefile v0.0.0-20221227161230-091c0ba34f0a // indirect
 	github.com/jackc/pgtype v1.14.0 // indirect
 	github.com/jackc/pgx/v4 v4.18.3 // indirect
-	github.com/libdns/libdns v0.2.2
+	github.com/libdns/libdns v0.2.3
 	github.com/manifoldco/promptui v0.9.0 // indirect
 	github.com/mattn/go-colorable v0.1.13 // indirect
 	github.com/mattn/go-isatty v0.0.20 // indirect
@@ -148,7 +149,7 @@ require (
 	go.uber.org/multierr v1.11.0 // indirect
 	golang.org/x/mod v0.18.0 // indirect
 	golang.org/x/sys v0.30.0
-	golang.org/x/text v0.21.0 // indirect
+	golang.org/x/text v0.22.0 // indirect
 	golang.org/x/tools v0.22.0 // indirect
 	google.golang.org/grpc v1.67.1 // indirect
 	google.golang.org/protobuf v1.35.1 // indirect
diff --git a/go.sum b/go.sum
index f9706143..ef978d02 100644
--- a/go.sum
+++ b/go.sum
@@ -91,8 +91,8 @@ github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM=
 github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw=
 github.com/bradfitz/go-smtpd v0.0.0-20170404230938-deb6d6237625/go.mod h1:HYsPBTaaSFSlLx/70C2HPIMNZpVV8+vt/A+FMnYP11g=
 github.com/buger/jsonparser v0.0.0-20181115193947-bf1c66bbce23/go.mod h1:bbYlZJ7hK1yFx9hf58LP0zeX7UjIGs20ufpu3evjr+s=
-github.com/caddyserver/certmagic v0.21.7 h1:66KJioPFJwttL43KYSWk7ErSmE6LfaJgCQuhm8Sg6fg=
-github.com/caddyserver/certmagic v0.21.7/go.mod h1:LCPG3WLxcnjVKl/xpjzM0gqh0knrKKKiO5WVttX2eEI=
+github.com/caddyserver/certmagic v0.21.8-0.20250220203412-a7894dd6992d h1:9zdfQHH838+rS8pmJ73/RSjpbfHGAyxRX1E79F+1zso=
+github.com/caddyserver/certmagic v0.21.8-0.20250220203412-a7894dd6992d/go.mod h1:LCPG3WLxcnjVKl/xpjzM0gqh0knrKKKiO5WVttX2eEI=
 github.com/caddyserver/zerossl v0.1.3 h1:onS+pxp3M8HnHpN5MMbOMyNjmTheJyWRaZYwn+YTAyA=
 github.com/caddyserver/zerossl v0.1.3/go.mod h1:CxA0acn7oEGO6//4rtrRjYgEoa4MFw/XofZnrYwGqG4=
 github.com/cenkalti/backoff/v4 v4.3.0 h1:MyRJ/UdXutAwSAT+s3wNd7MfTIcy71VQueUuFK343L8=
@@ -111,6 +111,8 @@ github.com/chzyer/test v0.0.0-20180213035817-a1ea475d72b1/go.mod h1:Q3SI9o4m/ZMn
 github.com/chzyer/test v1.0.0 h1:p3BQDXSxOhOG0P9z6/hGnII4LGiEPOYBhs8asl/fC04=
 github.com/chzyer/test v1.0.0/go.mod h1:2JlltgoNkt4TW/z9V/IzDdFaMTM2JPIi26O1pF38GC8=
 github.com/client9/misspell v0.3.4/go.mod h1:qj6jICC3Q7zFZvVWo7KLAzC3yx5G7kyvSDkc90ppPyw=
+github.com/cloudflare/circl v1.3.3 h1:fE/Qz0QdIGqeWfnwq0RE0R7MI51s0M2E4Ga9kq5AEMs=
+github.com/cloudflare/circl v1.3.3/go.mod h1:5XYMA4rFBvNIrhs50XuiBJ15vF2pZn4nnUKZrLbUZFA=
 github.com/cockroachdb/apd v1.1.0 h1:3LFP3629v+1aKXU5Q37mxmRxX/pIu1nijXydLShEq5I=
 github.com/cockroachdb/apd v1.1.0/go.mod h1:8Sl8LxpKi29FqWXR16WEFZRNSz3SoPzUzeMeY4+DwBQ=
 github.com/coreos/etcd v3.3.10+incompatible/go.mod h1:uF7uidLiAD3TWHmW31ZFd/JWoc32PjwdhPthX9715RE=
@@ -323,8 +325,8 @@ github.com/lib/pq v1.2.0/go.mod h1:5WUZQaWbwv1U+lTReE5YruASi9Al49XbQIvNi/34Woo=
 github.com/lib/pq v1.10.2/go.mod h1:AlVN5x4E4T544tWzH6hKfbfQvm3HdbOxrmggDNAPY9o=
 github.com/lib/pq v1.10.9 h1:YXG7RB+JIjhP29X+OtkiDnYaXQwpS4JEWq7dtCCRUEw=
 github.com/lib/pq v1.10.9/go.mod h1:AlVN5x4E4T544tWzH6hKfbfQvm3HdbOxrmggDNAPY9o=
-github.com/libdns/libdns v0.2.2 h1:O6ws7bAfRPaBsgAYt8MDe2HcNBGC29hkZ9MX2eUSX3s=
-github.com/libdns/libdns v0.2.2/go.mod h1:4Bj9+5CQiNMVGf87wjX4CY3HQJypUHRuLvlsfsZqLWQ=
+github.com/libdns/libdns v0.2.3 h1:ba30K4ObwMGB/QTmqUxf3H4/GmUrCAIkMWejeGl12v8=
+github.com/libdns/libdns v0.2.3/go.mod h1:4Bj9+5CQiNMVGf87wjX4CY3HQJypUHRuLvlsfsZqLWQ=
 github.com/lunixbochs/vtclean v1.0.0/go.mod h1:pHhQNgMf3btfWnGBVipUOjRYhoOsdGqdm/+2c2E2WMI=
 github.com/magiconair/properties v1.8.0/go.mod h1:PppfXfuXeibc/6YijjN8zIbojt8czPbwD3XqdrwzmxQ=
 github.com/mailru/easyjson v0.0.0-20190312143242-1de009706dbe/go.mod h1:C1wdFJiN94OJF2b5HbByQZoLdCWB1Yqtg26g4irojpc=
@@ -596,8 +598,8 @@ golang.org/x/crypto v0.0.0-20210616213533-5ff15b29337e/go.mod h1:GvvjBRRGRdwPK5y
 golang.org/x/crypto v0.0.0-20210711020723-a769d52b0f97/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc=
 golang.org/x/crypto v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc=
 golang.org/x/crypto v0.19.0/go.mod h1:Iy9bg/ha4yyC70EfRS8jz+B6ybOBKMaSxLj6P6oBDfU=
-golang.org/x/crypto v0.31.0 h1:ihbySMvVjLAeSH1IbfcRTkD/iNscyz8rGzjF/E5hV6U=
-golang.org/x/crypto v0.31.0/go.mod h1:kDsLvtWBEx7MV9tJOj9bnXsPbxwJQ6csT/x4KIN4Ssk=
+golang.org/x/crypto v0.33.0 h1:IOBPskki6Lysi0lo9qQvbxiQ+FvsCC/YWOecCHAixus=
+golang.org/x/crypto v0.33.0/go.mod h1:bVdXmD7IV/4GdElGPozy6U7lWdRXA4qyRVGJV57uQ5M=
 golang.org/x/crypto/x509roots/fallback v0.0.0-20241104001025-71ed71b4faf9 h1:4cEcP5+OjGppY79LCQ5Go2B1Boix2x0v6pvA01P3FoA=
 golang.org/x/crypto/x509roots/fallback v0.0.0-20241104001025-71ed71b4faf9/go.mod h1:kNa9WdvYnzFwC79zRpLRMJbdEFlhyM5RPFBBZp/wWH8=
 golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
@@ -645,8 +647,8 @@ golang.org/x/sync v0.0.0-20190227155943-e225da77a7e6/go.mod h1:RxMgew5VJxzue5/jJ
 golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
 golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
 golang.org/x/sync v0.1.0/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
-golang.org/x/sync v0.10.0 h1:3NQrjDixjgGwUOCaF8w2+VYHv0Ve/vGYSbdkTa98gmQ=
-golang.org/x/sync v0.10.0/go.mod h1:Czt+wKu1gCyEFDUtn0jG5QVvpJ6rzVqr5aXyt9drQfk=
+golang.org/x/sync v0.11.0 h1:GGz8+XQP4FvTTrjZPzNKTMFtSXH80RAzG+5ghFPgK9w=
+golang.org/x/sync v0.11.0/go.mod h1:Czt+wKu1gCyEFDUtn0jG5QVvpJ6rzVqr5aXyt9drQfk=
 golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
 golang.org/x/sys v0.0.0-20180905080454-ebe1bf3edb33/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
 golang.org/x/sys v0.0.0-20180909124046-d0be0721c37e/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
@@ -683,8 +685,8 @@ golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuX
 golang.org/x/term v0.5.0/go.mod h1:jMB1sMXY+tzblOD4FWmEbocvup2/aLOaQEp7JmGp78k=
 golang.org/x/term v0.8.0/go.mod h1:xPskH00ivmX89bAKVGSKKtLOWNx2+17Eiy94tnKShWo=
 golang.org/x/term v0.17.0/go.mod h1:lLRBjIVuehSbZlaOtGMbcMncT+aqLLLmKrsjNrUguwk=
-golang.org/x/term v0.27.0 h1:WP60Sv1nlK1T6SupCHbXzSaN0b9wUmsPoRS9b61A23Q=
-golang.org/x/term v0.27.0/go.mod h1:iMsnZpn0cago0GOrHO2+Y7u7JPn5AylBrcoWkElMTSM=
+golang.org/x/term v0.29.0 h1:L6pJp37ocefwRRtYPKSWOWzOtWSxVajvz2ldH/xi3iU=
+golang.org/x/term v0.29.0/go.mod h1:6bl4lRlvVuDgSf3179VpIxBF0o10JUpXWOnI7nErv7s=
 golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
 golang.org/x/text v0.3.1-0.20180807135948-17ff2d5776d2/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
 golang.org/x/text v0.3.2/go.mod h1:bEr9sfX3Q8Zfm5fL9x+3itogRgK3+ptLWKqgva+5dAk=
@@ -695,8 +697,8 @@ golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ=
 golang.org/x/text v0.7.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8=
 golang.org/x/text v0.9.0/go.mod h1:e1OnstbJyHTd6l/uOt8jFFHp6TRDWZR/bV3emEE/zU8=
 golang.org/x/text v0.14.0/go.mod h1:18ZOQIKpY8NJVqYksKHtTdi31H5itFRjB5/qKTNYzSU=
-golang.org/x/text v0.21.0 h1:zyQAAkrwaneQ066sspRyJaG9VNi/YJ1NfzcGB3hZ/qo=
-golang.org/x/text v0.21.0/go.mod h1:4IBbMaMmOPCJ8SecivzSH54+73PCFmPWxNTLm+vZkEQ=
+golang.org/x/text v0.22.0 h1:bofq7m3/HAFvbF51jz3Q9wLg3jkvSPuiZu/pD1XwgtM=
+golang.org/x/text v0.22.0/go.mod h1:YRoo4H8PVmsu+E3Ou7cqLVH8oXWIHVoX0jqUWALQhfY=
 golang.org/x/time v0.0.0-20180412165947-fbb02b2291d2/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
 golang.org/x/time v0.0.0-20181108054448-85acf8d2951c/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
 golang.org/x/time v0.7.0 h1:ntUhktv3OPE6TgYxXWv9vKvUSJyIFJlyohwbkEwPrKQ=
diff --git a/modules/caddyhttp/autohttps.go b/modules/caddyhttp/autohttps.go
index 4449e1f4..dce21a72 100644
--- a/modules/caddyhttp/autohttps.go
+++ b/modules/caddyhttp/autohttps.go
@@ -205,6 +205,7 @@ func (app *App) automaticHTTPSPhase1(ctx caddy.Context, repl *caddy.Replacer) er
 		// for all the hostnames we found, filter them so we have
 		// a deduplicated list of names for which to obtain certs
 		// (only if cert management not disabled for this server)
+		var echDomains []string
 		if srv.AutoHTTPS.DisableCerts {
 			logger.Warn("skipping automated certificate management for server because it is disabled", zap.String("server_name", srvName))
 		} else {
@@ -231,10 +232,14 @@ func (app *App) automaticHTTPSPhase1(ctx caddy.Context, repl *caddy.Replacer) er
 					}
 
 					uniqueDomainsForCerts[d] = struct{}{}
+					echDomains = append(echDomains, d)
 				}
 			}
 		}
 
+		// let the TLS server know we have some hostnames that could be protected behind ECH
+		app.tlsApp.RegisterServerNames(echDomains)
+
 		// tell the server to use TLS if it is not already doing so
 		if srv.TLSConnPolicies == nil {
 			srv.TLSConnPolicies = caddytls.ConnectionPolicies{new(caddytls.ConnectionPolicy)}
diff --git a/modules/caddytls/acmeissuer.go b/modules/caddytls/acmeissuer.go
index 2fe5eec9..c28790fe 100644
--- a/modules/caddytls/acmeissuer.go
+++ b/modules/caddytls/acmeissuer.go
@@ -146,15 +146,30 @@ func (iss *ACMEIssuer) Provision(ctx caddy.Context) error {
 		iss.AccountKey = accountKey
 	}
 
-	// DNS providers
-	if iss.Challenges != nil && iss.Challenges.DNS != nil && iss.Challenges.DNS.ProviderRaw != nil {
-		val, err := ctx.LoadModule(iss.Challenges.DNS, "ProviderRaw")
-		if err != nil {
-			return fmt.Errorf("loading DNS provider module: %v", err)
+	// DNS challenge provider
+	if iss.Challenges != nil && iss.Challenges.DNS != nil {
+		var prov certmagic.DNSProvider
+		if iss.Challenges.DNS.ProviderRaw != nil {
+			// a challenge provider has been locally configured - use it
+			val, err := ctx.LoadModule(iss.Challenges.DNS, "ProviderRaw")
+			if err != nil {
+				return fmt.Errorf("loading DNS provider module: %v", err)
+			}
+			prov = val.(certmagic.DNSProvider)
+		} else if tlsAppIface, err := ctx.AppIfConfigured("tls"); err == nil {
+			// no locally configured DNS challenge provider, but if there is
+			// a global DNS module configured with the TLS app, use that
+			tlsApp := tlsAppIface.(*TLS)
+			if tlsApp.dns != nil {
+				prov = tlsApp.dns.(certmagic.DNSProvider)
+			}
+		}
+		if prov == nil {
+			return fmt.Errorf("DNS challenge enabled, but no DNS provider configured")
 		}
 		iss.Challenges.DNS.solver = &certmagic.DNS01Solver{
 			DNSManager: certmagic.DNSManager{
-				DNSProvider:        val.(certmagic.DNSProvider),
+				DNSProvider:        prov,
 				TTL:                time.Duration(iss.Challenges.DNS.TTL),
 				PropagationDelay:   time.Duration(iss.Challenges.DNS.PropagationDelay),
 				PropagationTimeout: time.Duration(iss.Challenges.DNS.PropagationTimeout),
diff --git a/modules/caddytls/connpolicy.go b/modules/caddytls/connpolicy.go
index 727afaa0..b686090e 100644
--- a/modules/caddytls/connpolicy.go
+++ b/modules/caddytls/connpolicy.go
@@ -93,7 +93,7 @@ func (cp ConnectionPolicies) Provision(ctx caddy.Context) error {
 
 // TLSConfig returns a standard-lib-compatible TLS configuration which
 // selects the first matching policy based on the ClientHello.
-func (cp ConnectionPolicies) TLSConfig(_ caddy.Context) *tls.Config {
+func (cp ConnectionPolicies) TLSConfig(ctx caddy.Context) *tls.Config {
 	// using ServerName to match policies is extremely common, especially in configs
 	// with lots and lots of different policies; we can fast-track those by indexing
 	// them by SNI, so we don't have to iterate potentially thousands of policies
@@ -104,6 +104,7 @@ func (cp ConnectionPolicies) TLSConfig(_ caddy.Context) *tls.Config {
 			for _, m := range p.matchers {
 				if sni, ok := m.(MatchServerName); ok {
 					for _, sniName := range sni {
+						// index for fast lookups during handshakes
 						indexedBySNI[sniName] = append(indexedBySNI[sniName], p)
 					}
 				}
@@ -111,32 +112,79 @@ func (cp ConnectionPolicies) TLSConfig(_ caddy.Context) *tls.Config {
 		}
 	}
 
-	return &tls.Config{
-		MinVersion: tls.VersionTLS12,
-		GetConfigForClient: func(hello *tls.ClientHelloInfo) (*tls.Config, error) {
-			// filter policies by SNI first, if possible, to speed things up
-			// when there may be lots of policies
-			possiblePolicies := cp
-			if indexedPolicies, ok := indexedBySNI[hello.ServerName]; ok {
-				possiblePolicies = indexedPolicies
+	getConfigForClient := func(hello *tls.ClientHelloInfo) (*tls.Config, error) {
+		// filter policies by SNI first, if possible, to speed things up
+		// when there may be lots of policies
+		possiblePolicies := cp
+		if indexedPolicies, ok := indexedBySNI[hello.ServerName]; ok {
+			possiblePolicies = indexedPolicies
+		}
+
+	policyLoop:
+		for _, pol := range possiblePolicies {
+			for _, matcher := range pol.matchers {
+				if !matcher.Match(hello) {
+					continue policyLoop
+				}
 			}
+			if pol.Drop {
+				return nil, fmt.Errorf("dropping connection")
+			}
+			return pol.TLSConfig, nil
+		}
+
+		return nil, fmt.Errorf("no server TLS configuration available for ClientHello: %+v", hello)
+	}
 
-		policyLoop:
-			for _, pol := range possiblePolicies {
-				for _, matcher := range pol.matchers {
-					if !matcher.Match(hello) {
-						continue policyLoop
+	tlsCfg := &tls.Config{
+		MinVersion:         tls.VersionTLS12,
+		GetConfigForClient: getConfigForClient,
+	}
+
+	// enable ECH, if configured
+	if tlsAppIface, err := ctx.AppIfConfigured("tls"); err == nil {
+		tlsApp := tlsAppIface.(*TLS)
+
+		if tlsApp.EncryptedClientHello != nil && len(tlsApp.EncryptedClientHello.configs) > 0 {
+			// if no publication was configured, we apply ECH to all server names by default,
+			// but the TLS app needs to know what they are in this case, since they don't appear
+			// in its config (remember, TLS connection policies are used by *other* apps to
+			// run TLS servers) -- we skip names with placeholders
+			if tlsApp.EncryptedClientHello.Publication == nil {
+				var echNames []string
+				repl := caddy.NewReplacer()
+				for _, p := range cp {
+					for _, m := range p.matchers {
+						if sni, ok := m.(MatchServerName); ok {
+							for _, name := range sni {
+								finalName := strings.ToLower(repl.ReplaceAll(name, ""))
+								echNames = append(echNames, finalName)
+							}
+						}
 					}
 				}
-				if pol.Drop {
-					return nil, fmt.Errorf("dropping connection")
-				}
-				return pol.TLSConfig, nil
+				tlsApp.RegisterServerNames(echNames)
 			}
 
-			return nil, fmt.Errorf("no server TLS configuration available for ClientHello: %+v", hello)
-		},
+			// TODO: Ideally, ECH keys should be rotated. However, as of Go 1.24, the std lib implementation
+			// does not support safely modifying the tls.Config's EncryptedClientHelloKeys field.
+			// So, we implement static ECH keys temporarily. See https://github.com/golang/go/issues/71920.
+			// Revisit this after Go 1.25 is released and implement key rotation.
+			var stdECHKeys []tls.EncryptedClientHelloKey
+			for _, echConfigs := range tlsApp.EncryptedClientHello.configs {
+				for _, c := range echConfigs {
+					stdECHKeys = append(stdECHKeys, tls.EncryptedClientHelloKey{
+						Config:      c.configBin,
+						PrivateKey:  c.privKeyBin,
+						SendAsRetry: c.sendAsRetry,
+					})
+				}
+			}
+			tlsCfg.EncryptedClientHelloKeys = stdECHKeys
+		}
 	}
+
+	return tlsCfg
 }
 
 // ConnectionPolicy specifies the logic for handling a TLS handshake.
@@ -409,6 +457,7 @@ func (p ConnectionPolicy) SettingsEmpty() bool {
 		p.ProtocolMax == "" &&
 		p.ClientAuthentication == nil &&
 		p.DefaultSNI == "" &&
+		p.FallbackSNI == "" &&
 		p.InsecureSecretsLog == ""
 }
 
diff --git a/modules/caddytls/ech.go b/modules/caddytls/ech.go
new file mode 100644
index 00000000..05a375d9
--- /dev/null
+++ b/modules/caddytls/ech.go
@@ -0,0 +1,1074 @@
+package caddytls
+
+import (
+	"context"
+	"encoding/base64"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"io/fs"
+	weakrand "math/rand/v2"
+	"path"
+	"strconv"
+	"strings"
+	"time"
+
+	"github.com/caddyserver/certmagic"
+	"github.com/cloudflare/circl/hpke"
+	"github.com/cloudflare/circl/kem"
+	"github.com/libdns/libdns"
+	"go.uber.org/zap"
+	"golang.org/x/crypto/cryptobyte"
+
+	"github.com/caddyserver/caddy/v2"
+)
+
+func init() {
+	caddy.RegisterModule(ECHDNSPublisher{})
+}
+
+// ECH enables Encrypted ClientHello (ECH) and configures its management.
+//
+// Note that, as of Caddy 2.10 (~March 2025), ECH keys are not automatically
+// rotated due to a limitation in the Go standard library (see
+// https://github.com/golang/go/issues/71920). This should be resolved when
+// Go 1.25 is released (~Aug. 2025), and Caddy will be updated to automatically
+// rotate ECH keys/configs at that point.
+//
+// EXPERIMENTAL: Subject to change.
+type ECH struct {
+	// The list of ECH configurations for which to automatically generate
+	// and rotate keys. At least one is required to enable ECH.
+	Configs []ECHConfiguration `json:"configs,omitempty"`
+
+	// Publication describes ways to publish ECH configs for clients to
+	// discover and use. Without publication, most clients will not use
+	// ECH at all, and those that do will suffer degraded performance.
+	//
+	// Most major browsers support ECH by way of publication to HTTPS
+	// DNS RRs. (This also typically requires that they use DoH or DoT.)
+	Publication []*ECHPublication `json:"publication,omitempty"`
+
+	// map of public_name to list of configs
+	configs map[string][]echConfig
+}
+
+// Provision loads or creates ECH configs and returns outer names (for certificate
+// management), but does not publish any ECH configs. The DNS module is used as
+// a default for later publishing if needed.
+func (ech *ECH) Provision(ctx caddy.Context) ([]string, error) {
+	logger := ctx.Logger().Named("ech")
+
+	// set up publication modules before we need to obtain a lock in storage,
+	// since this is strictly internal and doesn't require synchronization
+	for i, pub := range ech.Publication {
+		mods, err := ctx.LoadModule(pub, "PublishersRaw")
+		if err != nil {
+			return nil, fmt.Errorf("loading ECH publication modules: %v", err)
+		}
+		for _, modIface := range mods.(map[string]any) {
+			ech.Publication[i].publishers = append(ech.Publication[i].publishers, modIface.(ECHPublisher))
+		}
+	}
+
+	// the rest of provisioning needs an exclusive lock so that instances aren't
+	// stepping on each other when setting up ECH configs
+	storage := ctx.Storage()
+	const echLockName = "ech_provision"
+	if err := storage.Lock(ctx, echLockName); err != nil {
+		return nil, err
+	}
+	defer func() {
+		if err := storage.Unlock(ctx, echLockName); err != nil {
+			logger.Error("unable to unlock ECH provisioning in storage", zap.Error(err))
+		}
+	}()
+
+	var outerNames []string //nolint:prealloc // (FALSE POSITIVE - see https://github.com/alexkohler/prealloc/issues/30)
+
+	// start by loading all the existing configs (even the older ones on the way out,
+	// since some clients may still be using them if they haven't yet picked up on the
+	// new configs)
+	cfgKeys, err := storage.List(ctx, echConfigsKey, false)
+	if err != nil && !errors.Is(err, fs.ErrNotExist) { // OK if dir doesn't exist; it will be created
+		return nil, err
+	}
+	for _, cfgKey := range cfgKeys {
+		cfg, err := loadECHConfig(ctx, path.Base(cfgKey))
+		if err != nil {
+			return nil, err
+		}
+		// if any part of the config's folder was corrupted, the load function will
+		// clean it up and not return an error, since configs are immutable and
+		// fairly ephemeral... so just check that we actually got a populated config
+		if cfg.configBin == nil || cfg.privKeyBin == nil {
+			continue
+		}
+		logger.Debug("loaded ECH config",
+			zap.String("public_name", cfg.RawPublicName),
+			zap.Uint8("id", cfg.ConfigID))
+		ech.configs[cfg.RawPublicName] = append(ech.configs[cfg.RawPublicName], cfg)
+		outerNames = append(outerNames, cfg.RawPublicName)
+	}
+
+	// all existing configs are now loaded; see if we need to make any new ones
+	// based on the input configuration, and also mark the most recent one(s) as
+	// current/active, so they can be used for ECH retries
+
+	for _, cfg := range ech.Configs {
+		publicName := strings.ToLower(strings.TrimSpace(cfg.OuterSNI))
+
+		if list, ok := ech.configs[publicName]; ok && len(list) > 0 {
+			// at least one config with this public name was loaded, so find the
+			// most recent one and mark it as active to be used with retries
+			var mostRecentDate time.Time
+			var mostRecentIdx int
+			for i, c := range list {
+				if mostRecentDate.IsZero() || c.meta.Created.After(mostRecentDate) {
+					mostRecentDate = c.meta.Created
+					mostRecentIdx = i
+				}
+			}
+			list[mostRecentIdx].sendAsRetry = true
+		} else {
+			// no config with this public name was loaded, so create one
+			echCfg, err := generateAndStoreECHConfig(ctx, publicName)
+			if err != nil {
+				return nil, err
+			}
+			logger.Debug("generated new ECH config",
+				zap.String("public_name", echCfg.RawPublicName),
+				zap.Uint8("id", echCfg.ConfigID))
+			ech.configs[publicName] = append(ech.configs[publicName], echCfg)
+			outerNames = append(outerNames, publicName)
+		}
+	}
+
+	return outerNames, nil
+}
+
+func (t *TLS) publishECHConfigs() error {
+	logger := t.logger.Named("ech")
+
+	// make publication exclusive, since we don't need to repeat this unnecessarily
+	storage := t.ctx.Storage()
+	const echLockName = "ech_publish"
+	if err := storage.Lock(t.ctx, echLockName); err != nil {
+		return err
+	}
+	defer func() {
+		if err := storage.Unlock(t.ctx, echLockName); err != nil {
+			logger.Error("unable to unlock ECH provisioning in storage", zap.Error(err))
+		}
+	}()
+
+	// get the publication config, or use a default if not specified
+	// (the default publication config should be to publish all ECH
+	// configs to the app-global DNS provider; if no DNS provider is
+	// configured, then this whole function is basically a no-op)
+	publicationList := t.EncryptedClientHello.Publication
+	if publicationList == nil {
+		if dnsProv, ok := t.dns.(ECHDNSProvider); ok {
+			publicationList = []*ECHPublication{
+				{
+					publishers: []ECHPublisher{
+						&ECHDNSPublisher{
+							provider: dnsProv,
+							logger:   t.logger,
+						},
+					},
+				},
+			}
+		}
+	}
+
+	// for each publication config, build the list of ECH configs to
+	// publish with it, and figure out which inner names to publish
+	// to/for, then publish
+	for _, publication := range publicationList {
+		// this publication is either configured for specific ECH configs,
+		// or we just use an implied default of all ECH configs
+		var echCfgList echConfigList
+		var configIDs []uint8 // TODO: use IDs or the outer names?
+		if publication.Configs == nil {
+			// by default, publish all configs
+			for _, configs := range t.EncryptedClientHello.configs {
+				echCfgList = append(echCfgList, configs...)
+				for _, c := range configs {
+					configIDs = append(configIDs, c.ConfigID)
+				}
+			}
+		} else {
+			for _, cfgOuterName := range publication.Configs {
+				if cfgList, ok := t.EncryptedClientHello.configs[cfgOuterName]; ok {
+					echCfgList = append(echCfgList, cfgList...)
+					for _, c := range cfgList {
+						configIDs = append(configIDs, c.ConfigID)
+					}
+				}
+			}
+		}
+
+		// marshal the ECH config list as binary for publication
+		echCfgListBin, err := echCfgList.MarshalBinary()
+		if err != nil {
+			return fmt.Errorf("marshaling ECH config list: %v", err)
+		}
+
+		// now we have our list of ECH configs to publish and the inner names
+		// to publish for (i.e. the names being protected); iterate each publisher
+		// and do the publish for any config+name that needs a publish
+		for _, publisher := range publication.publishers {
+			publisherKey := publisher.PublisherKey()
+
+			// by default, publish for all (non-outer) server names, unless
+			// a specific list of names is configured
+			var serverNamesSet map[string]struct{}
+			if publication.Domains == nil {
+				serverNamesSet = make(map[string]struct{}, len(t.serverNames))
+				for name := range t.serverNames {
+					serverNamesSet[name] = struct{}{}
+				}
+			} else {
+				serverNamesSet = make(map[string]struct{}, len(publication.Domains))
+				for _, name := range publication.Domains {
+					serverNamesSet[name] = struct{}{}
+				}
+			}
+
+			// remove any domains from the set which have already had all configs in the
+			// list published by this publisher, to avoid always re-publishing unnecessarily
+			for configuredInnerName := range serverNamesSet {
+				allConfigsPublished := true
+				for _, cfg := range echCfgList {
+					// TODO: Potentially utilize the timestamp (map value) for recent-enough publication, instead of just checking for existence
+					if _, ok := cfg.meta.Publications[publisherKey][configuredInnerName]; !ok {
+						allConfigsPublished = false
+						break
+					}
+				}
+				if allConfigsPublished {
+					delete(serverNamesSet, configuredInnerName)
+				}
+			}
+
+			// if all the (inner) domains have had this ECH config list published
+			// by this publisher, then try the next publication config
+			if len(serverNamesSet) == 0 {
+				logger.Debug("ECH config list already published by publisher for associated domains",
+					zap.Uint8s("config_ids", configIDs),
+					zap.String("publisher", publisherKey))
+				continue
+			}
+
+			// convert the set of names to a slice
+			dnsNamesToPublish := make([]string, 0, len(serverNamesSet))
+			for name := range serverNamesSet {
+				dnsNamesToPublish = append(dnsNamesToPublish, name)
+			}
+
+			logger.Debug("publishing ECH config list",
+				zap.Strings("domains", dnsNamesToPublish),
+				zap.Uint8s("config_ids", configIDs))
+
+			// publish this ECH config list with this publisher
+			pubTime := time.Now()
+			err := publisher.PublishECHConfigList(t.ctx, dnsNamesToPublish, echCfgListBin)
+			if err != nil {
+				t.logger.Error("publishing ECH configuration list",
+					zap.Strings("for_domains", publication.Domains),
+					zap.Error(err))
+			}
+
+			// update publication history, so that we don't unnecessarily republish every time
+			for _, cfg := range echCfgList {
+				if cfg.meta.Publications == nil {
+					cfg.meta.Publications = make(publicationHistory)
+				}
+				if _, ok := cfg.meta.Publications[publisherKey]; !ok {
+					cfg.meta.Publications[publisherKey] = make(map[string]time.Time)
+				}
+				for _, name := range dnsNamesToPublish {
+					cfg.meta.Publications[publisherKey][name] = pubTime
+				}
+				metaBytes, err := json.Marshal(cfg.meta)
+				if err != nil {
+					return fmt.Errorf("marshaling ECH config metadata: %v", err)
+				}
+				metaKey := path.Join(echConfigsKey, strconv.Itoa(int(cfg.ConfigID)), "meta.json")
+				if err := t.ctx.Storage().Store(t.ctx, metaKey, metaBytes); err != nil {
+					return fmt.Errorf("storing updated ECH config metadata: %v", err)
+				}
+			}
+		}
+	}
+
+	return nil
+}
+
+// loadECHConfig loads the config from storage with the given configID.
+// An error is not actually returned in some cases the config fails to
+// load because in some cases it just means the config ID folder has
+// been cleaned up in storage, maybe due to an incomplete set of keys
+// or corrupted contents; in any case, the only rectification is to
+// delete it and make new keys (an error IS returned if deleting the
+// corrupted keys fails, for example). Check the returned echConfig for
+// non-nil privKeyBin and configBin values before using.
+func loadECHConfig(ctx caddy.Context, configID string) (echConfig, error) {
+	storage := ctx.Storage()
+	logger := ctx.Logger()
+
+	cfgIDKey := path.Join(echConfigsKey, configID)
+	keyKey := path.Join(cfgIDKey, "key.bin")
+	configKey := path.Join(cfgIDKey, "config.bin")
+	metaKey := path.Join(cfgIDKey, "meta.json")
+
+	// if loading anything fails, might as well delete this folder and free up
+	// the config ID; spec is designed to rotate configs frequently anyway
+	// (I consider it a more serious error if we can't clean up the folder,
+	// since leaving stray storage keys is confusing)
+	privKeyBytes, err := storage.Load(ctx, keyKey)
+	if err != nil {
+		delErr := storage.Delete(ctx, cfgIDKey)
+		if delErr != nil {
+			return echConfig{}, fmt.Errorf("error loading private key (%v) and cleaning up parent storage key %s: %v", err, cfgIDKey, delErr)
+		}
+		logger.Warn("could not load ECH private key; deleting its config folder",
+			zap.String("config_id", configID),
+			zap.Error(err))
+		return echConfig{}, nil
+	}
+	echConfigBytes, err := storage.Load(ctx, configKey)
+	if err != nil {
+		delErr := storage.Delete(ctx, cfgIDKey)
+		if delErr != nil {
+			return echConfig{}, fmt.Errorf("error loading ECH config (%v) and cleaning up parent storage key %s: %v", err, cfgIDKey, delErr)
+		}
+		logger.Warn("could not load ECH config; deleting its config folder",
+			zap.String("config_id", configID),
+			zap.Error(err))
+		return echConfig{}, nil
+	}
+	var cfg echConfig
+	if err := cfg.UnmarshalBinary(echConfigBytes); err != nil {
+		delErr := storage.Delete(ctx, cfgIDKey)
+		if delErr != nil {
+			return echConfig{}, fmt.Errorf("error loading ECH config (%v) and cleaning up parent storage key %s: %v", err, cfgIDKey, delErr)
+		}
+		logger.Warn("could not load ECH config; deleted its config folder",
+			zap.String("config_id", configID),
+			zap.Error(err))
+		return echConfig{}, nil
+	}
+	metaBytes, err := storage.Load(ctx, metaKey)
+	if err != nil {
+		delErr := storage.Delete(ctx, cfgIDKey)
+		if delErr != nil {
+			return echConfig{}, fmt.Errorf("error loading ECH metadata (%v) and cleaning up parent storage key %s: %v", err, cfgIDKey, delErr)
+		}
+		logger.Warn("could not load ECH metadata; deleted its config folder",
+			zap.String("config_id", configID),
+			zap.Error(err))
+		return echConfig{}, nil
+	}
+	var meta echConfigMeta
+	if err := json.Unmarshal(metaBytes, &meta); err != nil {
+		// even though it's just metadata, reset the whole config since we can't reliably maintain it
+		delErr := storage.Delete(ctx, cfgIDKey)
+		if delErr != nil {
+			return echConfig{}, fmt.Errorf("error decoding ECH metadata (%v) and cleaning up parent storage key %s: %v", err, cfgIDKey, delErr)
+		}
+		logger.Warn("could not JSON-decode ECH metadata; deleted its config folder",
+			zap.String("config_id", configID),
+			zap.Error(err))
+		return echConfig{}, nil
+	}
+
+	cfg.privKeyBin = privKeyBytes
+	cfg.configBin = echConfigBytes
+	cfg.meta = meta
+
+	return cfg, nil
+}
+
+func generateAndStoreECHConfig(ctx caddy.Context, publicName string) (echConfig, error) {
+	// Go currently has very strict requirements for server-side ECH configs,
+	// to quote the Go 1.24 godoc (with typos of AEAD IDs corrected):
+	//
+	// "Config should be a marshalled ECHConfig associated with PrivateKey. This
+	// must match the config provided to clients byte-for-byte. The config
+	// should only specify the DHKEM(X25519, HKDF-SHA256) KEM ID (0x0020), the
+	// HKDF-SHA256 KDF ID (0x0001), and a subset of the following AEAD IDs:
+	// AES-128-GCM (0x0001), AES-256-GCM (0x0002), ChaCha20Poly1305 (0x0003)."
+	//
+	// So we need to be sure we generate a config within these parameters
+	// so the Go TLS server can use it.
+
+	// generate a key pair
+	const kemChoice = hpke.KEM_X25519_HKDF_SHA256
+	publicKey, privateKey, err := kemChoice.Scheme().GenerateKeyPair()
+	if err != nil {
+		return echConfig{}, err
+	}
+
+	// find an available config ID
+	configID, err := newECHConfigID(ctx)
+	if err != nil {
+		return echConfig{}, fmt.Errorf("generating unique config ID: %v", err)
+	}
+
+	echCfg := echConfig{
+		PublicKey:     publicKey,
+		Version:       draftTLSESNI22,
+		ConfigID:      configID,
+		RawPublicName: publicName,
+		KEMID:         kemChoice,
+		CipherSuites: []hpkeSymmetricCipherSuite{
+			{
+				KDFID:  hpke.KDF_HKDF_SHA256,
+				AEADID: hpke.AEAD_AES128GCM,
+			},
+			{
+				KDFID:  hpke.KDF_HKDF_SHA256,
+				AEADID: hpke.AEAD_AES256GCM,
+			},
+			{
+				KDFID:  hpke.KDF_HKDF_SHA256,
+				AEADID: hpke.AEAD_ChaCha20Poly1305,
+			},
+		},
+		sendAsRetry: true,
+	}
+	meta := echConfigMeta{
+		Created: time.Now(),
+	}
+
+	privKeyBytes, err := privateKey.MarshalBinary()
+	if err != nil {
+		return echConfig{}, fmt.Errorf("marshaling ECH private key: %v", err)
+	}
+	echConfigBytes, err := echCfg.MarshalBinary()
+	if err != nil {
+		return echConfig{}, fmt.Errorf("marshaling ECH config: %v", err)
+	}
+	metaBytes, err := json.Marshal(meta)
+	if err != nil {
+		return echConfig{}, fmt.Errorf("marshaling ECH config metadata: %v", err)
+	}
+
+	parentKey := path.Join(echConfigsKey, strconv.Itoa(int(configID)))
+	keyKey := path.Join(parentKey, "key.bin")
+	configKey := path.Join(parentKey, "config.bin")
+	metaKey := path.Join(parentKey, "meta.json")
+
+	if err := ctx.Storage().Store(ctx, keyKey, privKeyBytes); err != nil {
+		return echConfig{}, fmt.Errorf("storing ECH private key: %v", err)
+	}
+	if err := ctx.Storage().Store(ctx, configKey, echConfigBytes); err != nil {
+		return echConfig{}, fmt.Errorf("storing ECH config: %v", err)
+	}
+	if err := ctx.Storage().Store(ctx, metaKey, metaBytes); err != nil {
+		return echConfig{}, fmt.Errorf("storing ECH config metadata: %v", err)
+	}
+
+	echCfg.privKeyBin = privKeyBytes
+	echCfg.configBin = echConfigBytes // this contains the public key
+	echCfg.meta = meta
+
+	return echCfg, nil
+}
+
+// ECH represents an Encrypted ClientHello configuration.
+//
+// EXPERIMENTAL: Subject to change.
+type ECHConfiguration struct {
+	// The public server name that will be used in the outer ClientHello. This
+	// should be a domain name for which this server is authoritative, because
+	// Caddy will try to provision a certificate for this name. As an outer
+	// SNI, it is never used for application data (HTTPS, etc.), but it is
+	// necessary for securely reconciling inconsistent client state without
+	// breakage and brittleness.
+	OuterSNI string `json:"outer_sni,omitempty"`
+}
+
+// ECHPublication configures publication of ECH config(s).
+type ECHPublication struct {
+	// TODO: Should these first two fields be called outer_sni and inner_sni ?
+
+	// The list of ECH configurations to publish, identified by public name.
+	// If not set, all configs will be included for publication by default.
+	Configs []string `json:"configs,omitempty"`
+
+	// The list of domain names which are protected with the associated ECH
+	// configurations ("inner names"). Not all publishers may require this
+	// information, but some, like the DNS publisher, do. (The DNS publisher,
+	// for example, needs to know for which domain(s) to create DNS records.)
+	//
+	// If not set, all server names registered with the TLS module will be
+	// added to this list implicitly. (Other Caddy apps that use the TLS
+	// module automatically register their configured server names for this
+	// purpose. For example, the HTTP server registers the hostnames for
+	// which it applies automatic HTTPS.)
+	//
+	// Names in this list should not appear in any other publication config
+	// object with the same publishers, since the publications will likely
+	// overwrite each other.
+	//
+	// NOTE: In order to publish ECH configs for domains configured for
+	// On-Demand TLS that are not explicitly enumerated elsewhere in the
+	// config, those domain names will have to be listed here. The only
+	// time Caddy knows which domains it is serving with On-Demand TLS is
+	// handshake-time, which is too late for publishing ECH configs; it
+	// means the first connections would not protect the server names,
+	// revealing that information to observers, and thus defeating the
+	// purpose of ECH. Hence the need to list them here so Caddy can
+	// proactively publish ECH configs before clients connect with those
+	// server names in plaintext.
+	Domains []string `json:"domains,omitempty"`
+
+	// How to publish the ECH configurations so clients can know to use them.
+	// Note that ECH configs are only published when they are newly created,
+	// so adding or changing publishers after the fact will have no effect
+	// with existing ECH configs. The next time a config is generated (including
+	// when a key is rotated), the current publication modules will be utilized.
+	PublishersRaw caddy.ModuleMap `json:"publishers,omitempty" caddy:"namespace=tls.ech.publishers"`
+	publishers    []ECHPublisher
+}
+
+// ECHDNSProvider can service DNS entries for ECH purposes.
+type ECHDNSProvider interface {
+	libdns.RecordGetter
+	libdns.RecordSetter
+}
+
+// ECHDNSPublisher configures how to publish an ECH configuration to
+// DNS records for the specified domains.
+//
+// EXPERIMENTAL: Subject to change.
+type ECHDNSPublisher struct {
+	// The DNS provider module which will establish the HTTPS record(s).
+	ProviderRaw json.RawMessage `json:"provider,omitempty" caddy:"namespace=dns.providers inline_key=name"`
+	provider    ECHDNSProvider
+
+	logger *zap.Logger
+}
+
+// CaddyModule returns the Caddy module information.
+func (ECHDNSPublisher) CaddyModule() caddy.ModuleInfo {
+	return caddy.ModuleInfo{
+		ID:  "tls.ech.publishers.dns",
+		New: func() caddy.Module { return new(ECHDNSPublisher) },
+	}
+}
+
+func (dnsPub ECHDNSPublisher) Provision(ctx caddy.Context) error {
+	dnsProvMod, err := ctx.LoadModule(dnsPub, "ProviderRaw")
+	if err != nil {
+		return fmt.Errorf("loading ECH DNS provider module: %v", err)
+	}
+	prov, ok := dnsProvMod.(ECHDNSProvider)
+	if !ok {
+		return fmt.Errorf("ECH DNS provider module is not an ECH DNS Provider: %v", err)
+	}
+	dnsPub.provider = prov
+	dnsPub.logger = ctx.Logger()
+	return nil
+}
+
+// PublisherKey returns the name of the DNS provider module.
+// We intentionally omit specific provider configuration (or a hash thereof,
+// since the config is likely sensitive, potentially containing an API key)
+// because it is unlikely that specific configuration, such as an API key,
+// is relevant to unique key use as an ECH config publisher.
+func (dnsPub ECHDNSPublisher) PublisherKey() string {
+	return string(dnsPub.provider.(caddy.Module).CaddyModule().ID)
+}
+
+// PublishECHConfigList publishes the given ECH config list to the given DNS names.
+func (dnsPub *ECHDNSPublisher) PublishECHConfigList(ctx context.Context, innerNames []string, configListBin []byte) error {
+	nameservers := certmagic.RecursiveNameservers(nil) // TODO: we could make resolvers configurable
+
+	for _, domain := range innerNames {
+		zone, err := certmagic.FindZoneByFQDN(ctx, dnsPub.logger, domain, nameservers)
+		if err != nil {
+			dnsPub.logger.Error("could not determine zone for domain",
+				zap.String("domain", domain),
+				zap.Error(err))
+			continue
+		}
+
+		// get any existing HTTPS record for this domain, and augment
+		// our ech SvcParamKey with any other existing SvcParams
+		recs, err := dnsPub.provider.GetRecords(ctx, zone)
+		if err != nil {
+			dnsPub.logger.Error("unable to get existing DNS records to publish ECH data to HTTPS DNS record",
+				zap.String("domain", domain),
+				zap.Error(err))
+			continue
+		}
+		relName := libdns.RelativeName(domain+".", zone)
+		var httpsRec libdns.Record
+		for _, rec := range recs {
+			if rec.Name == relName && rec.Type == "HTTPS" && (rec.Target == "" || rec.Target == ".") {
+				httpsRec = rec
+			}
+		}
+		params := make(svcParams)
+		if httpsRec.Value != "" {
+			params, err = parseSvcParams(httpsRec.Value)
+			if err != nil {
+				dnsPub.logger.Error("unable to parse existing DNS record to publish ECH data to HTTPS DNS record",
+					zap.String("domain", domain),
+					zap.String("https_rec_value", httpsRec.Value),
+					zap.Error(err))
+				continue
+			}
+		}
+
+		// overwrite only the ech SvcParamKey
+		params["ech"] = []string{base64.StdEncoding.EncodeToString(configListBin)}
+
+		// publish record
+		_, err = dnsPub.provider.SetRecords(ctx, zone, []libdns.Record{
+			{
+				// HTTPS and SVCB RRs: RFC 9460 (https://www.rfc-editor.org/rfc/rfc9460)
+				Type:     "HTTPS",
+				Name:     relName,
+				Priority: 2, // allows a manual override with priority 1
+				Target:   ".",
+				Value:    params.String(),
+				TTL:      1 * time.Minute, // TODO: for testing only
+			},
+		})
+		if err != nil {
+			dnsPub.logger.Error("unable to publish ECH data to HTTPS DNS record",
+				zap.String("domain", domain),
+				zap.Error(err))
+			continue
+		}
+	}
+
+	return nil
+}
+
+// echConfig represents an ECHConfig from the specification,
+// [draft-ietf-tls-esni-22](https://www.ietf.org/archive/id/draft-ietf-tls-esni-22.html).
+type echConfig struct {
+	// "The version of ECH for which this configuration is used.
+	// The version is the same as the code point for the
+	// encrypted_client_hello extension. Clients MUST ignore any
+	// ECHConfig structure with a version they do not support."
+	Version uint16
+
+	// The "length" and "contents" fields defined next in the
+	// structure are implicitly taken care of by cryptobyte
+	// when encoding the following fields:
+
+	// HpkeKeyConfig fields:
+	ConfigID     uint8
+	KEMID        hpke.KEM
+	PublicKey    kem.PublicKey
+	CipherSuites []hpkeSymmetricCipherSuite
+
+	// ECHConfigContents fields:
+	MaxNameLength uint8
+	RawPublicName string
+	RawExtensions []byte
+
+	// these fields are not part of the spec, but are here for
+	// our use when setting up TLS servers or maintenance
+	configBin   []byte
+	privKeyBin  []byte
+	meta        echConfigMeta
+	sendAsRetry bool
+}
+
+func (echCfg echConfig) MarshalBinary() ([]byte, error) {
+	var b cryptobyte.Builder
+	if err := echCfg.marshalBinary(&b); err != nil {
+		return nil, err
+	}
+	return b.Bytes()
+}
+
+// UnmarshalBinary decodes the data back into an ECH config.
+//
+// Borrowed from github.com/OmarTariq612/goech with modifications.
+// Original code: Copyright (c) 2023 Omar Tariq AbdEl-Raziq
+func (echCfg *echConfig) UnmarshalBinary(data []byte) error {
+	var content cryptobyte.String
+	b := cryptobyte.String(data)
+
+	if !b.ReadUint16(&echCfg.Version) {
+		return errInvalidLen
+	}
+	if echCfg.Version != draftTLSESNI22 {
+		return fmt.Errorf("supported version must be %d: got %d", draftTLSESNI22, echCfg.Version)
+	}
+
+	if !b.ReadUint16LengthPrefixed(&content) || !b.Empty() {
+		return errInvalidLen
+	}
+
+	var t cryptobyte.String
+	var pk []byte
+
+	if !content.ReadUint8(&echCfg.ConfigID) ||
+		!content.ReadUint16((*uint16)(&echCfg.KEMID)) ||
+		!content.ReadUint16LengthPrefixed(&t) ||
+		!t.ReadBytes(&pk, len(t)) ||
+		!content.ReadUint16LengthPrefixed(&t) ||
+		len(t)%4 != 0 /* the length of (KDFs and AEADs) must be divisible by 4 */ {
+		return errInvalidLen
+	}
+
+	if !echCfg.KEMID.IsValid() {
+		return fmt.Errorf("invalid KEM ID: %d", echCfg.KEMID)
+	}
+
+	var err error
+	if echCfg.PublicKey, err = echCfg.KEMID.Scheme().UnmarshalBinaryPublicKey(pk); err != nil {
+		return fmt.Errorf("parsing public_key: %w", err)
+	}
+
+	echCfg.CipherSuites = echCfg.CipherSuites[:0]
+
+	for !t.Empty() {
+		var hpkeKDF, hpkeAEAD uint16
+		if !t.ReadUint16(&hpkeKDF) || !t.ReadUint16(&hpkeAEAD) {
+			// we have already checked that the length is divisible by 4
+			panic("this must not happen")
+		}
+		if !hpke.KDF(hpkeKDF).IsValid() {
+			return fmt.Errorf("invalid KDF ID: %d", hpkeKDF)
+		}
+		if !hpke.AEAD(hpkeAEAD).IsValid() {
+			return fmt.Errorf("invalid AEAD ID: %d", hpkeAEAD)
+		}
+		echCfg.CipherSuites = append(echCfg.CipherSuites, hpkeSymmetricCipherSuite{
+			KDFID:  hpke.KDF(hpkeKDF),
+			AEADID: hpke.AEAD(hpkeAEAD),
+		})
+	}
+
+	var rawPublicName []byte
+	if !content.ReadUint8(&echCfg.MaxNameLength) ||
+		!content.ReadUint8LengthPrefixed(&t) ||
+		!t.ReadBytes(&rawPublicName, len(t)) ||
+		!content.ReadUint16LengthPrefixed(&t) ||
+		!t.ReadBytes(&echCfg.RawExtensions, len(t)) ||
+		!content.Empty() {
+		return errInvalidLen
+	}
+	echCfg.RawPublicName = string(rawPublicName)
+
+	return nil
+}
+
+var errInvalidLen = errors.New("invalid length")
+
+// marshalBinary writes this config to the cryptobyte builder. If there is an error,
+// it will occur before any writes have happened.
+func (echCfg echConfig) marshalBinary(b *cryptobyte.Builder) error {
+	pk, err := echCfg.PublicKey.MarshalBinary()
+	if err != nil {
+		return err
+	}
+	if l := len(echCfg.RawPublicName); l == 0 || l > 255 {
+		return fmt.Errorf("public name length (%d) must be in the range 1-255", l)
+	}
+
+	b.AddUint16(echCfg.Version)
+	b.AddUint16LengthPrefixed(func(b *cryptobyte.Builder) { // "length" field
+		b.AddUint8(echCfg.ConfigID)
+		b.AddUint16(uint16(echCfg.KEMID))
+		b.AddUint16LengthPrefixed(func(b *cryptobyte.Builder) {
+			b.AddBytes(pk)
+		})
+		b.AddUint16LengthPrefixed(func(b *cryptobyte.Builder) {
+			for _, cs := range echCfg.CipherSuites {
+				b.AddUint16(uint16(cs.KDFID))
+				b.AddUint16(uint16(cs.AEADID))
+			}
+		})
+		b.AddUint8(uint8(min(len(echCfg.RawPublicName)+16, 255)))
+		b.AddUint8LengthPrefixed(func(b *cryptobyte.Builder) {
+			b.AddBytes([]byte(echCfg.RawPublicName))
+		})
+		b.AddUint16LengthPrefixed(func(child *cryptobyte.Builder) {
+			child.AddBytes(echCfg.RawExtensions)
+		})
+	})
+
+	return nil
+}
+
+type hpkeSymmetricCipherSuite struct {
+	KDFID  hpke.KDF
+	AEADID hpke.AEAD
+}
+
+type echConfigList []echConfig
+
+func (cl echConfigList) MarshalBinary() ([]byte, error) {
+	var b cryptobyte.Builder
+	var err error
+
+	// the list's length prefixes the list, as with most opaque values
+	b.AddUint16LengthPrefixed(func(b *cryptobyte.Builder) {
+		for _, cfg := range cl {
+			if err = cfg.marshalBinary(b); err != nil {
+				break
+			}
+		}
+	})
+	if err != nil {
+		return nil, err
+	}
+
+	return b.Bytes()
+}
+
+func newECHConfigID(ctx caddy.Context) (uint8, error) {
+	// uint8 can be 0-255 inclusive
+	const uint8Range = 256
+
+	// avoid repeating storage checks
+	tried := make([]bool, uint8Range)
+
+	// Try to find an available number with random rejection sampling;
+	// i.e. choose a random number and see if it's already taken.
+	// The hard limit on how many times we try to find an available
+	// number is flexible... in theory, assuming uniform distribution,
+	// 256 attempts should make each possible value show up exactly
+	// once, but obviously that won't be the case. We can try more
+	// times to try to ensure that every number gets a chance, which
+	// is especially useful if few are available, or we can lower it
+	// if we assume we should have found an available value by then
+	// and want to limit runtime; for now I choose the middle ground
+	// and just try as many times as there are possible values.
+	for i := 0; i < uint8Range && ctx.Err() == nil; i++ {
+		num := uint8(weakrand.N(uint8Range)) //nolint:gosec
+
+		// don't try the same number a second time
+		if tried[num] {
+			continue
+		}
+		tried[num] = true
+
+		// check to see if any of the subkeys use this config ID
+		numStr := strconv.Itoa(int(num))
+		trialPath := path.Join(echConfigsKey, numStr)
+		if ctx.Storage().Exists(ctx, trialPath) {
+			continue
+		}
+
+		return num, nil
+	}
+
+	if err := ctx.Err(); err != nil {
+		return 0, err
+	}
+
+	return 0, fmt.Errorf("depleted attempts to find an available config_id")
+}
+
+// svcParams represents SvcParamKey and SvcParamValue pairs as
+// described in https://www.rfc-editor.org/rfc/rfc9460 (section 2.1).
+type svcParams map[string][]string
+
+// parseSvcParams parses service parameters into a structured type
+// for safer manipulation.
+func parseSvcParams(input string) (svcParams, error) {
+	if len(input) > 4096 {
+		return nil, fmt.Errorf("input too long: %d", len(input))
+	}
+
+	params := make(svcParams)
+	input = strings.TrimSpace(input) + " "
+
+	for cursor := 0; cursor < len(input); cursor++ {
+		var key, rawVal string
+
+	keyValPair:
+		for i := cursor; i < len(input); i++ {
+			switch input[i] {
+			case '=':
+				key = strings.ToLower(strings.TrimSpace(input[cursor:i]))
+				i++
+				cursor = i
+
+				var quoted bool
+				if input[cursor] == '"' {
+					quoted = true
+					i++
+					cursor = i
+				}
+
+				var escaped bool
+
+				for j := cursor; j < len(input); j++ {
+					switch input[j] {
+					case '"':
+						if !quoted {
+							return nil, fmt.Errorf("illegal DQUOTE at position %d", j)
+						}
+						if !escaped {
+							// end of quoted value
+							rawVal = input[cursor:j]
+							j++
+							cursor = j
+							break keyValPair
+						}
+					case '\\':
+						escaped = true
+					case ' ', '\t', '\n', '\r':
+						if !quoted {
+							// end of unquoted value
+							rawVal = input[cursor:j]
+							cursor = j
+							break keyValPair
+						}
+					default:
+						escaped = false
+					}
+				}
+
+			case ' ', '\t', '\n', '\r':
+				// key with no value (flag)
+				key = input[cursor:i]
+				params[key] = []string{}
+				cursor = i
+				break keyValPair
+			}
+		}
+
+		if rawVal == "" {
+			continue
+		}
+
+		var sb strings.Builder
+
+		var escape int // start of escape sequence (after \, so 0 is never a valid start)
+		for i := 0; i < len(rawVal); i++ {
+			ch := rawVal[i]
+			if escape > 0 {
+				// validate escape sequence
+				// (RFC 9460 Appendix A)
+				// escaped:   "\" ( non-digit / dec-octet )
+				// non-digit: "%x21-2F / %x3A-7E"
+				// dec-octet: "0-255 as a 3-digit decimal number"
+				if ch >= '0' && ch <= '9' {
+					// advance to end of decimal octet, which must be 3 digits
+					i += 2
+					if i > len(rawVal) {
+						return nil, fmt.Errorf("value ends with incomplete escape sequence: %s", rawVal[escape:])
+					}
+					decOctet, err := strconv.Atoi(rawVal[escape : i+1])
+					if err != nil {
+						return nil, err
+					}
+					if decOctet < 0 || decOctet > 255 {
+						return nil, fmt.Errorf("invalid decimal octet in escape sequence: %s (%d)", rawVal[escape:i], decOctet)
+					}
+					sb.WriteRune(rune(decOctet))
+					escape = 0
+					continue
+				} else if (ch < 0x21 || ch > 0x2F) && (ch < 0x3A && ch > 0x7E) {
+					return nil, fmt.Errorf("illegal escape sequence %s", rawVal[escape:i])
+				}
+			}
+			switch ch {
+			case ';', '(', ')':
+				// RFC 9460 Appendix A:
+				// > contiguous  = 1*( non-special / escaped )
+				// > non-special is VCHAR minus DQUOTE, ";", "(", ")", and "\".
+				return nil, fmt.Errorf("illegal character in value %q at position %d: %s", rawVal, i, string(ch))
+			case '\\':
+				escape = i + 1
+			default:
+				sb.WriteByte(ch)
+				escape = 0
+			}
+		}
+
+		params[key] = strings.Split(sb.String(), ",")
+	}
+
+	return params, nil
+}
+
+// String serializes svcParams into zone presentation format.
+func (params svcParams) String() string {
+	var sb strings.Builder
+	for key, vals := range params {
+		if sb.Len() > 0 {
+			sb.WriteRune(' ')
+		}
+		sb.WriteString(key)
+		var hasVal, needsQuotes bool
+		for _, val := range vals {
+			if len(val) > 0 {
+				hasVal = true
+			}
+			if strings.ContainsAny(val, `" `) {
+				needsQuotes = true
+			}
+			if hasVal && needsQuotes {
+				break
+			}
+		}
+		if hasVal {
+			sb.WriteRune('=')
+		}
+		if needsQuotes {
+			sb.WriteRune('"')
+		}
+		for i, val := range vals {
+			if i > 0 {
+				sb.WriteRune(',')
+			}
+			val = strings.ReplaceAll(val, `"`, `\"`)
+			val = strings.ReplaceAll(val, `,`, `\,`)
+			sb.WriteString(val)
+		}
+		if needsQuotes {
+			sb.WriteRune('"')
+		}
+	}
+	return sb.String()
+}
+
+// ECHPublisher is an interface for publishing ECHConfigList values
+// so that they can be used by clients.
+type ECHPublisher interface {
+	// Returns a key that is unique to this publisher and its configuration.
+	// A publisher's ID combined with its config is a valid key.
+	// It is used to prevent duplicating publications.
+	PublisherKey() string
+
+	// Publishes the ECH config list for the given innerNames. Some publishers
+	// may not need a list of inner/protected names, and can ignore the argument;
+	// most, however, will want to use it to know which inner names are to be
+	// associated with the given ECH config list.
+	PublishECHConfigList(ctx context.Context, innerNames []string, echConfigList []byte) error
+}
+
+type echConfigMeta struct {
+	Created      time.Time          `json:"created"`
+	Publications publicationHistory `json:"publications"`
+}
+
+// publicationHistory is a map of publisher key to
+// map of inner name to timestamp
+type publicationHistory map[string]map[string]time.Time
+
+// The key prefix when putting ECH configs in storage. After this
+// comes the config ID.
+const echConfigsKey = "ech/configs"
+
+// https://www.ietf.org/archive/id/draft-ietf-tls-esni-22.html
+const draftTLSESNI22 = 0xfe0d
+
+// Interface guard
+var _ ECHPublisher = (*ECHDNSPublisher)(nil)
diff --git a/modules/caddytls/tls.go b/modules/caddytls/tls.go
index abb519eb..423a2f9a 100644
--- a/modules/caddytls/tls.go
+++ b/modules/caddytls/tls.go
@@ -20,12 +20,15 @@ import (
 	"encoding/json"
 	"fmt"
 	"log"
+	"net"
 	"net/http"
 	"runtime/debug"
+	"strings"
 	"sync"
 	"time"
 
 	"github.com/caddyserver/certmagic"
+	"github.com/libdns/libdns"
 	"go.uber.org/zap"
 	"go.uber.org/zap/zapcore"
 
@@ -79,6 +82,7 @@ type TLS struct {
 	// Disabling OCSP stapling puts clients at greater risk, reduces their
 	// privacy, and usually lowers client performance. It is NOT recommended
 	// to disable this unless you are able to justify the costs.
+	//
 	// EXPERIMENTAL. Subject to change.
 	DisableOCSPStapling bool `json:"disable_ocsp_stapling,omitempty"`
 
@@ -89,6 +93,7 @@ type TLS struct {
 	//
 	// Disabling these checks should only be done when the storage
 	// can be trusted to have enough capacity and no other problems.
+	//
 	// EXPERIMENTAL. Subject to change.
 	DisableStorageCheck bool `json:"disable_storage_check,omitempty"`
 
@@ -100,9 +105,23 @@ type TLS struct {
 	// The instance.uuid file is used to identify the instance of Caddy
 	// in a cluster. The last_clean.json file is used to store the last
 	// time the storage was cleaned.
+	//
 	// EXPERIMENTAL. Subject to change.
 	DisableStorageClean bool `json:"disable_storage_clean,omitempty"`
 
+	// Enable Encrypted ClientHello (ECH). ECH protects the server name
+	// (SNI) and other sensitive parameters of a normally-plaintext TLS
+	// ClientHello during a handshake.
+	//
+	// EXPERIMENTAL: Subject to change.
+	EncryptedClientHello *ECH `json:"encrypted_client_hello,omitempty"`
+
+	// The default DNS provider module to use when a DNS module is needed.
+	//
+	// EXPERIMENTAL: Subject to change.
+	DNSRaw json.RawMessage `json:"dns,omitempty" caddy:"namespace=dns.providers inline_key=name"`
+	dns    any             // technically, it should be any/all of the libdns interfaces (RecordSetter, RecordAppender, etc.)
+
 	certificateLoaders []CertificateLoader
 	automateNames      []string
 	ctx                caddy.Context
@@ -111,6 +130,9 @@ type TLS struct {
 	logger             *zap.Logger
 	events             *caddyevents.App
 
+	serverNames   map[string]struct{}
+	serverNamesMu *sync.Mutex
+
 	// set of subjects with managed certificates,
 	// and hashes of manually-loaded certificates
 	// (managing's value is an optional issuer key, for distinction)
@@ -136,6 +158,40 @@ func (t *TLS) Provision(ctx caddy.Context) error {
 	t.logger = ctx.Logger()
 	repl := caddy.NewReplacer()
 	t.managing, t.loaded = make(map[string]string), make(map[string]string)
+	t.serverNames = make(map[string]struct{})
+	t.serverNamesMu = new(sync.Mutex)
+
+	// set up default DNS module, if any, and make sure it implements all the
+	// common libdns interfaces, since it could be used for a variety of things
+	// (do this before provisioning other modules, since they may rely on this)
+	if len(t.DNSRaw) > 0 {
+		dnsMod, err := ctx.LoadModule(t, "DNSRaw")
+		if err != nil {
+			return fmt.Errorf("loading overall DNS provider module: %v", err)
+		}
+		switch dnsMod.(type) {
+		case interface {
+			libdns.RecordAppender
+			libdns.RecordDeleter
+			libdns.RecordGetter
+			libdns.RecordSetter
+		}:
+		default:
+			return fmt.Errorf("DNS module does not implement the most common libdns interfaces: %T", dnsMod)
+		}
+		t.dns = dnsMod
+	}
+
+	// ECH (Encrypted ClientHello) initialization
+	if t.EncryptedClientHello != nil {
+		t.EncryptedClientHello.configs = make(map[string][]echConfig)
+		outerNames, err := t.EncryptedClientHello.Provision(ctx)
+		if err != nil {
+			return fmt.Errorf("provisioning Encrypted ClientHello components: %v", err)
+		}
+		// outer names should have certificates to reduce client brittleness
+		t.automateNames = append(t.automateNames, outerNames...)
+	}
 
 	// set up a new certificate cache; this (re)loads all certificates
 	cacheOpts := certmagic.CacheOptions{
@@ -178,7 +234,7 @@ func (t *TLS) Provision(ctx caddy.Context) error {
 				for i, sub := range *automateNames {
 					subjects[i] = repl.ReplaceAll(sub, "")
 				}
-				t.automateNames = subjects
+				t.automateNames = append(t.automateNames, subjects...)
 			} else {
 				return fmt.Errorf("loading certificates with 'automate' requires array of strings, got: %T", modIface)
 			}
@@ -339,6 +395,16 @@ func (t *TLS) Start() error {
 		return fmt.Errorf("automate: managing %v: %v", t.automateNames, err)
 	}
 
+	// publish ECH configs in the background; does not need to block
+	// server startup, as it could take a while
+	if t.EncryptedClientHello != nil {
+		go func() {
+			if err := t.publishECHConfigs(); err != nil {
+				t.logger.Named("ech").Error("publication(s) failed", zap.Error(err))
+			}
+		}()
+	}
+
 	if !t.DisableStorageClean {
 		// start the storage cleaner goroutine and ticker,
 		// which cleans out expired certificates and more
@@ -422,11 +488,16 @@ func (t *TLS) Cleanup() error {
 			}
 		}
 	} else {
-		// no more TLS app running, so delete in-memory cert cache
-		certCache.Stop()
-		certCacheMu.Lock()
-		certCache = nil
-		certCacheMu.Unlock()
+		// no more TLS app running, so delete in-memory cert cache, if it was created yet
+		certCacheMu.RLock()
+		hasCache := certCache != nil
+		certCacheMu.RUnlock()
+		if hasCache {
+			certCache.Stop()
+			certCacheMu.Lock()
+			certCache = nil
+			certCacheMu.Unlock()
+		}
 	}
 
 	return nil
@@ -478,6 +549,29 @@ func (t *TLS) Manage(names []string) error {
 	return nil
 }
 
+// RegisterServerNames registers the provided DNS names with the TLS app.
+// This is currently used to auto-publish Encrypted ClientHello (ECH)
+// configurations, if enabled. Use of this function by apps using the TLS
+// app removes the need for the user to redundantly specify domain names
+// in their configuration. This function separates hostname and port
+// (keeping only the hotsname) and filters IP addresses, which can't be
+// used with ECH.
+//
+// EXPERIMENTAL: This function and its behavior are subject to change.
+func (t *TLS) RegisterServerNames(dnsNames []string) {
+	t.serverNamesMu.Lock()
+	for _, name := range dnsNames {
+		host, _, err := net.SplitHostPort(name)
+		if err != nil {
+			host = name
+		}
+		if strings.TrimSpace(host) != "" && !certmagic.SubjectIsIP(host) {
+			t.serverNames[strings.ToLower(host)] = struct{}{}
+		}
+	}
+	t.serverNamesMu.Unlock()
+}
+
 // HandleHTTPChallenge ensures that the ACME HTTP challenge or ZeroSSL HTTP
 // validation request is handled for the certificate named by r.Host, if it
 // is an HTTP challenge request. It requires that the automation policy for



Test patch:
diff --git a/modules/caddytls/ech_test.go b/modules/caddytls/ech_test.go
new file mode 100644
index 00000000..b722d2fb
--- /dev/null
+++ b/modules/caddytls/ech_test.go
@@ -0,0 +1,129 @@
+package caddytls
+
+import (
+	"reflect"
+	"testing"
+)
+
+func TestParseSvcParams(t *testing.T) {
+	for i, test := range []struct {
+		input     string
+		expect    svcParams
+		shouldErr bool
+	}{
+		{
+			input: `alpn="h2,h3" no-default-alpn ipv6hint=2001:db8::1 port=443`,
+			expect: svcParams{
+				"alpn":            {"h2", "h3"},
+				"no-default-alpn": {},
+				"ipv6hint":        {"2001:db8::1"},
+				"port":            {"443"},
+			},
+		},
+		{
+			input: `key=value quoted="some string" flag`,
+			expect: svcParams{
+				"key":    {"value"},
+				"quoted": {"some string"},
+				"flag":   {},
+			},
+		},
+		{
+			input: `key="nested \"quoted\" value,foobar"`,
+			expect: svcParams{
+				"key": {`nested "quoted" value`, "foobar"},
+			},
+		},
+		{
+			input: `alpn=h3,h2 tls-supported-groups=29,23 no-default-alpn ech="foobar"`,
+			expect: svcParams{
+				"alpn":                 {"h3", "h2"},
+				"tls-supported-groups": {"29", "23"},
+				"no-default-alpn":      {},
+				"ech":                  {"foobar"},
+			},
+		},
+		{
+			input: `escape=\097`,
+			expect: svcParams{
+				"escape": {"a"},
+			},
+		},
+		{
+			input: `escapes=\097\098c`,
+			expect: svcParams{
+				"escapes": {"abc"},
+			},
+		},
+	} {
+		actual, err := parseSvcParams(test.input)
+		if err != nil && !test.shouldErr {
+			t.Errorf("Test %d: Expected no error, but got: %v (input=%q)", i, err, test.input)
+			continue
+		} else if err == nil && test.shouldErr {
+			t.Errorf("Test %d: Expected an error, but got no error (input=%q)", i, test.input)
+			continue
+		}
+		if !reflect.DeepEqual(test.expect, actual) {
+			t.Errorf("Test %d: Expected %v, got %v (input=%q)", i, test.expect, actual, test.input)
+			continue
+		}
+	}
+}
+
+func TestSvcParamsString(t *testing.T) {
+	// this test relies on the parser also working
+	// because we can't just compare string outputs
+	// since map iteration is unordered
+	for i, test := range []svcParams{
+
+		{
+			"alpn":            {"h2", "h3"},
+			"no-default-alpn": {},
+			"ipv6hint":        {"2001:db8::1"},
+			"port":            {"443"},
+		},
+
+		{
+			"key":    {"value"},
+			"quoted": {"some string"},
+			"flag":   {},
+		},
+		{
+			"key": {`nested "quoted" value`, "foobar"},
+		},
+		{
+			"alpn":                 {"h3", "h2"},
+			"tls-supported-groups": {"29", "23"},
+			"no-default-alpn":      {},
+			"ech":                  {"foobar"},
+		},
+	} {
+		combined := test.String()
+		parsed, err := parseSvcParams(combined)
+		if err != nil {
+			t.Errorf("Test %d: Expected no error, but got: %v (input=%q)", i, err, test)
+			continue
+		}
+		if len(parsed) != len(test) {
+			t.Errorf("Test %d: Expected %d keys, but got %d", i, len(test), len(parsed))
+			continue
+		}
+		for key, expectedVals := range test {
+			if expected, actual := len(expectedVals), len(parsed[key]); expected != actual {
+				t.Errorf("Test %d: Expected key %s to have %d values, but had %d", i, key, expected, actual)
+				continue
+			}
+			for j, expected := range expectedVals {
+				if actual := parsed[key][j]; actual != expected {
+					t.Errorf("Test %d key %q value %d: Expected '%s' but got '%s'", i, key, j, expected, actual)
+					continue
+				}
+			}
+		}
+		if !reflect.DeepEqual(parsed, test) {
+			t.Errorf("Test %d: Expected %#v, got %#v", i, test, combined)
+			continue
+		}
+	}
+}