Repository: caddyserver/caddy (language: go)

Issue:
trusted_ca_cert_file should support multiple trusted CA certs in one file
When working with trust bundles and mutual TLS, we need to rotate CA certs which means running new and old CAs in parallel for a while. This requires a Caddy to accept client TLS certs with multiple CAs.

Currently `trusted_ca_cert_file` does not allow multiple CA certs in one file, we have to have separate tooling to split CA trust bundles into separate files and generate Caddyfiles with multiple directives from templates. All our other tooling at this point is happy with everything in one file except Caddy.

Is this a conscious design / security descision with a rationale that I can't figure out or something that could be improved?



Discussion / hints:
I don't think we've had a request for that before. So do you mean a single file that has multiple trust chains? (I'm surprised that doesn't work already.)
I was surprised also. Maybe I'm doing something wrong but here is a test setup:

## Generate 2x CA and mTLS (client + server auth) certs

### mtls.conf 
```
[client]
basicConstraints = CA:FALSE
nsCertType = client, server
nsComment = "mTLS test"
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid,issuer
keyUsage = critical, nonRepudiation, digitalSignature, keyEncipherment
extendedKeyUsage = clientAuth, serverAuth
```

### OpenSSL magic
```
for ca in ca1 ca2; do mkdir -p $ca; openssl genrsa -out $ca/$ca.key 2048; openssl req -x509 -batch -new -nodes -key $ca/$ca.key -sha256 -days 300 -out $ca/$ca.crt; openssl genrsa -out $ca/$ca-mtls.key 4096; openssl req -new -key $ca/$ca-mtls.key -out $ca/$ca-mtls.csr -sha256 -subj "/CN=$ca client"; openssl x509 -req -days 300 -in $ca/$ca-mtls.csr -sha256 -CA $ca/$ca.crt -CAkey $ca/$ca.key -CAcreateserial -out $ca/$ca-mtls.crt -extfile mtls.conf; done
```

## Combine both CA certs into a bundle

```
cat ca?/ca?.crt > ca_bundle.pem
```

## Caddyfiles

### Caddyfile.separate
```
{
    debug
    auto_https off
}

:443 {
    tls /etc/caddy/ca1/ca1-mtls.crt /etc/caddy/ca1/ca1-mtls.key {
        client_auth {
            mode                  require_and_verify
            trusted_ca_cert_file  /etc/caddy/ca1/ca1.crt
            trusted_ca_cert_file  /etc/caddy/ca2/ca2.crt
        }
    }
    respond "Separate OK"
}
```

### Caddyfile.bundle

```
{
    debug
    auto_https off
}

:444 {
    tls /etc/caddy/ca1/ca1-mtls.crt /etc/caddy/ca1/ca1-mtls.key {
        client_auth {
            mode                  require_and_verify
            trusted_ca_cert_file  /etc/caddy/ca_bundle.pem
        }
    }
    respond "Bundle OK"
}
```

## Start Caddy containers

```
docker run --rm -d --net host --name caddy_separate -v $PWD:/etc/caddy caddy:latest caddy run --config /etc/caddy/Caddyfile.separate
docker run --rm -d --net host --name caddy_bundle -v $PWD:/etc/caddy caddy:latest caddy run --config /etc/caddy/Caddyfile.bundle
```

## Test both mTLS client certs against both Caddy with cURL

```
for ca in ca1 ca2; do curl -k --cert $ca/$ca-mtls.crt --key $ca/$ca-mtls.key https://localhost:443/; echo; done
curl: (56) OpenSSL SSL_read: OpenSSL/3.0.11: error:0A000418:SSL routines::tlsv1 alert unknown ca, errno 0
curl: (56) OpenSSL SSL_read: OpenSSL/3.0.11: error:0A000418:SSL routines::tlsv1 alert unknown ca, errno 0
```
Result: Both client certs fail against bundle-Caddy

```
for ca in ca1 ca2; do curl -k --cert $ca/$ca-mtls.crt --key $ca/$ca-mtls.key https://localhost:444/; echo; done
Separate OK
Separate OK
```
Result: Noth client certs are ok against "normal" Caddy

## Clean UP
```
docker stop caddy_bundle caddy_separate
```

Caddy version in the test (via `:latest`) was `v2.7.6 h1:w0NymbG2m9PcvKWsrXO6EEkY9Ru4FJK8uQbYcev1p3A=`
What do you get in Caddy's debug logs?
@francislavoie here you go:

CA Bundle Caddy config:
```javascript
{"level":"info","ts":1703057961.5260248,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile.bundle","config_adapter":""}
...
{"level":"debug","ts":1703057961.5587604,"logger":"http.auto_https","msg":"adjusted config","tls":{"automation":{"policies":[{}]}},"http":{"servers":{"srv0":{"listen":[":443"],"routes":[{"handle":[{"body":"Bundle OK","handler":"static_response"}]}],"tls_connection_policies":[{"certificate_selection":{"any_tag":["cert0"]},"client_authentication":{"trusted_ca_certs":["MIIDazCCAlOgAwIBAgIUWnFev2hRuoEl0fb3ZPjzabNu9BwwDQYJKoZIhvcNAQELBQAwRTELMAkGA1UEBhMCQVUxEzARBgNVBAgMClNvbWUtU3RhdGUxITAfBgNVBAoMGEludGVybmV0IFdpZGdpdHMgUHR5IEx0ZDAeFw0yMzEyMTkxMjM4MzBaFw0yNDAxMTgxMjM4MzBaMEUxCzAJBgNVBAYTAkFVMRMwEQYDVQQIDApTb21lLVN0YXRlMSEwHwYDVQQKDBhJbnRlcm5ldCBXaWRnaXRzIFB0eSBMdGQwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC7oOEMto5pDcB9B9KswjWaGyLSdsfC3Gx+FX0nozVOBF2GQOdps3/HBnT7HIV8WZFTtnaMJqe0klVilzaL20XOnyhvmm/7fHHJq+qRvlY0WWeoGSVBm5CrGhurgWNEHc8KT6V3Cp6KAQcOVsjpSe19hXB4Q1ZJB+JMRAa3fN6XeLNzAHxHbNkLsEEuKGd0+N00qB2zw0J/7D3/jnUmzag2OtVnDLMp1nInmQ63iZFqiXnz0VTdvuUpjgodqTQAVsiW4QeywQBUYCMn98bBYDhr3zkVO9IkTDi+b+DtOYa6WfU/vfKLkqSZuAjxOK7RNPivq0Yp3/bZdFrcZMSbwKaDAgMBAAGjUzBRMB0GA1UdDgQWBBS8GNu/ApAj61cX7uYDl4Y27aPgSTAfBgNVHSMEGDAWgBS8GNu/ApAj61cX7uYDl4Y27aPgSTAPBgNVHRMBAf8EBTADAQH/MA0GCSqGSIb3DQEBCwUAA4IBAQA4ZBtzgjXmjuNuL4Op6m9D76cbAMAs74iPwUH8iouYnkPgUKy88bMLnoTfas+FJx7YVi/elYS8WXZpMI+Rww5gcqih26S3vLdsLAfb2MfCjfG43+4HYlFO2GTp9I8OjddDdZSoBr4Tav8LqIuGiyoU4rGwymIMSrXpZ/LVTuh4UpJxS3tae4Fy1a1fMP7yTUfTPSJAaqPjH3//mVo9+3h2gusOurLGmOOGSzIGdzlvsfMEQDkOALBpIlc746wUA/4+RrMi34B0cOzmOr1mGCPN3DBmFl+ttqcJn2XPxPCmhMSchSXEMLG61XAkoLLIzdQ75PWwIzC25gRjoQpMRdPO"],"mode":"require_and_verify"}}],"automatic_https":{"disable":true}}}}}
...
```
Note that `trusted_ca_certs` list has only one element.

Bundle Caddy failed request:
```javascript
{"level":"debug","ts":1703057971.2294648,"logger":"events","msg":"event","name":"tls_get_certificate","id":"5d219180-ad8c-4646-8611-22c0cc65bafd","origin":"tls","data":{"client_hello":{"CipherSuites":[4866,4867,4865,49196,49200,159,52393,52392,52394,49195,49199,158,49188,49192,107,49187,49191,103,49162,49172,57,49161,49171,51,157,156,61,60,53,47,255],"ServerName":"localhost","SupportedCurves":[29,23,30,25,24,256,257,258,259,260],"SupportedPoints":"AAEC","SignatureSchemes":[1027,1283,1539,2055,2056,2057,2058,2059,2052,2053,2054,1025,1281,1537,771,769,770,1026,1282,1538],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[772,771,770,769],"RemoteAddr":{"IP":"127.0.0.1","Port":44898,"Zone":""},"LocalAddr":{"IP":"127.0.0.1","Port":443,"Zone":""}}}}
{"level":"debug","ts":1703057971.2297914,"logger":"tls.handshake","msg":"no matching certificate; will choose from all certificates","identifier":"localhost"}
{"level":"debug","ts":1703057971.2298033,"logger":"tls.handshake","msg":"choosing certificate","identifier":"localhost","num_choices":1}
{"level":"debug","ts":1703057971.2299218,"logger":"tls.handshake","msg":"custom certificate selection results","identifier":"localhost","subjects":["ca1 client"],"managed":false,"issuer_key":"","hash":"698fb9489431bcc67ac926b90dfeb62613b774afacfcc03730c23044cea23c73"}
{"level":"debug","ts":1703057971.2299466,"logger":"tls.handshake","msg":"matched certificate in cache","remote_ip":"127.0.0.1","remote_port":"44898","subjects":["ca1 client"],"managed":false,"expiration":1705581914,"hash":"698fb9489431bcc67ac926b90dfeb62613b774afacfcc03730c23044cea23c73"}
{"level":"debug","ts":1703057971.2738655,"logger":"http.stdlib","msg":"http: TLS handshake error from 127.0.0.1:44898: tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"Internet Widgits Pty Ltd\")"}
```


Separate-Caddy config:
```javascript
{"level":"info","ts":1703057961.1946905,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile.separate","config_adapter":""}
...
{"level":"debug","ts":1703057961.2021763,"logger":"http.auto_https","msg":"adjusted config","tls":{"automation":{"policies":[{}]}},"http":{"servers":{"srv0":{"listen":[":444"],"routes":[{"handle":[{"body":"Separate OK","handler":"static_response"}]}],"tls_connection_policies":[{"certificate_selection":{"any_tag":["cert0"]},"client_authentication":{"trusted_ca_certs":["MIIDazCCAlOgAwIBAgIUfOSzZPI8yGXhMh1IejN0PPWy6zswDQYJKoZIhvcNAQELBQAwRTELMAkGA1UEBhMCQVUxEzARBgNVBAgMClNvbWUtU3RhdGUxITAfBgNVBAoMGEludGVybmV0IFdpZGdpdHMgUHR5IEx0ZDAeFw0yMzEyMTkxMjQ1MTBaFw0yNDAxMTgxMjQ1MTBaMEUxCzAJBgNVBAYTAkFVMRMwEQYDVQQIDApTb21lLVN0YXRlMSEwHwYDVQQKDBhJbnRlcm5ldCBXaWRnaXRzIFB0eSBMdGQwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCxQnSi98WCSAnPqrLZnR4XNm3tQyS3+25CiPYECBxY9tCd9x6S4CQ/CI+jUa7stQ9Rs/o+r3J+DvgguIwmoRjzaL+0qrbHGmq7LNVleSsaKlFwN+v52ef4vgteliKHw61sz1/bRnu2wVaPF20SKEtcBzoMAGnvSVJperiW1YAX9fEwGPg7ds7P/7/VHk0D5O6F7wNAgcBdmHsskXr3BKSatwahZ7ynMtWURtzdIvy1BrsYyS3tyQ5XKhnTzcf032SobwfdPnU3TTHMTfK/r7oZEV52iKRB5oeRNGHAloh5uFIVN0al+cuflATluXkIWJgj9ppe0trsDavCm7K1VfslAgMBAAGjUzBRMB0GA1UdDgQWBBT91O746P/nQxkACuPNPc30q1b6EDAfBgNVHSMEGDAWgBT91O746P/nQxkACuPNPc30q1b6EDAPBgNVHRMBAf8EBTADAQH/MA0GCSqGSIb3DQEBCwUAA4IBAQAU0vcF9QWoPoNLXbbJZYha/WQbjN3KF1raK9pxCPhFYuFXku5mQ0lFpvak2m6eUjF94sPW7VxfzRR4iA9x1Q6xmcbUoWi2JN7I9iPbijTPscGAccQNJnrXFaFWTlqr4BdOnYDmpF1cyihdkgQGIsdxkwLY1kFJl9s3AkaNwRS2YG/TMoR6tbk5msLtLEghgzlpM33+Q5GMe+QRmSaPyktSFouABZGVtUJb2G9PTcGuhJmq0M/2clWNa59hZgsrZPA+qoYpCF4AUn8PllugWylyS5kc0uAfaGopT6qqTPWwzN87vtEhr0bRlmMakxuXOTT03HXNfomHAnMqNBMwI3OC","MIIDazCCAlOgAwIBAgIUcOf/nAbxdRE8meGTwatUHm5HpwUwDQYJKoZIhvcNAQELBQAwRTELMAkGA1UEBhMCQVUxEzARBgNVBAgMClNvbWUtU3RhdGUxITAfBgNVBAoMGEludGVybmV0IFdpZGdpdHMgUHR5IEx0ZDAeFw0yMzEyMTkxMjQ1MTRaFw0yNDAxMTgxMjQ1MTRaMEUxCzAJBgNVBAYTAkFVMRMwEQYDVQQIDApTb21lLVN0YXRlMSEwHwYDVQQKDBhJbnRlcm5ldCBXaWRnaXRzIFB0eSBMdGQwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDHlttheHtWgIPuNJasCqr6jRFp6xygP5/H41RaooehraLk6kBKDrb9tCcYWk2nK/HQMO25b7qAtgm42idCoSQrQeYYJ2D+4f7Ye74Tf1ZT9K9Yu5lPa+0Z8EybbVLS9+rGICuMqE1qG7flJQG1fqmZjbAJBRKlTMBBlxDeFyo4bj7O0FKfbU0Jvvknk9Q6VLVj3QLSsz9U55pHLYiHg2KohdGrvmIqZCon0SmADLmwTwFXKaw+cthrNJZlkvVcrKD4mt6GA7wwBxi+t6j8SgIEtTK5svDgeuAQt5QU7c7qTVkUv2oNpgKsUoh6CPpZE2wULKx7BH1hinZqaEqoBWT9AgMBAAGjUzBRMB0GA1UdDgQWBBRpR230gyqwONq29XSq4KVQba+DazAfBgNVHSMEGDAWgBRpR230gyqwONq29XSq4KVQba+DazAPBgNVHRMBAf8EBTADAQH/MA0GCSqGSIb3DQEBCwUAA4IBAQCbwNIL7OXklLdXbpTeRB4mPDAhBXgCNL/lnTh3oa10cGi2ptA5bPuzipprg0W5eKlXYFDsFJS0dIlBitQ5KS/sAPB8AOE67sEkwm5fD0584tqEItqbXBgUlO6NeVd91Ncx9JJdMmvnyAYdCl0kj8S2RQmIPhxFTM313FNxhEIRXX/h+Ahrrws8aBv7q3cga8CzNEO6IN1keeA67PChCO7NIii/YmaQAMD+TSpYbpmMqYAIiWncsKmgVkwbMZAj8cC5kb01ipHJ974oaayHLQLtPBoUDDdTk3eN5r4qY63rq/5M50OuIBP2ChGkk+VCxM9wa5nA7bYhUacbiDefAwWY"],"mode":"require_and_verify"}}],"automatic_https":{"disable":true}}}}}
...
```
And unlike with the bundle, `trusted_ca_certs` list now has TWO elements...

Separate-Caddy OK request:
```javascript
{"level":"debug","ts":1703058541.5328152,"logger":"events","msg":"event","name":"tls_get_certificate","id":"ab11b519-db71-4a10-8d92-5665efb79843","origin":"tls","data":{"client_hello":{"CipherSuites":[4866,4867,4865,49196,49200,159,52393,52392,52394,49195,49199,158,49188,49192,107,49187,49191,103,49162,49172,57,49161,49171,51,157,156,61,60,53,47,255],"ServerName":"localhost","SupportedCurves":[29,23,30,25,24,256,257,258,259,260],"SupportedPoints":"AAEC","SignatureSchemes":[1027,1283,1539,2055,2056,2057,2058,2059,2052,2053,2054,1025,1281,1537,771,769,770,1026,1282,1538],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[772,771,770,769],"RemoteAddr":{"IP":"127.0.0.1","Port":37344,"Zone":""},"LocalAddr":{"IP":"127.0.0.1","Port":444,"Zone":""}}}}
{"level":"debug","ts":1703058541.533096,"logger":"tls.handshake","msg":"no matching certificate; will choose from all certificates","identifier":"localhost"}
{"level":"debug","ts":1703058541.533118,"logger":"tls.handshake","msg":"choosing certificate","identifier":"localhost","num_choices":1}
{"level":"debug","ts":1703058541.5332243,"logger":"tls.handshake","msg":"custom certificate selection results","identifier":"localhost","subjects":["ca1 client"],"managed":false,"issuer_key":"","hash":"698fb9489431bcc67ac926b90dfeb62613b774afacfcc03730c23044cea23c73"}
{"level":"debug","ts":1703058541.5332499,"logger":"tls.handshake","msg":"matched certificate in cache","remote_ip":"127.0.0.1","remote_port":"37344","subjects":["ca1 client"],"managed":false,"expiration":1705581914,"hash":"698fb9489431bcc67ac926b90dfeb62613b774afacfcc03730c23044cea23c73"}
{"level":"debug","ts":1703058541.5991879,"logger":"events","msg":"event","name":"tls_get_certificate","id":"ac63f74f-d24a-46ab-bcb8-0545d11ba367","origin":"tls","data":{"client_hello":{"CipherSuites":[4866,4867,4865,49196,49200,159,52393,52392,52394,49195,49199,158,49188,49192,107,49187,49191,103,49162,49172,57,49161,49171,51,157,156,61,60,53,47,255],"ServerName":"localhost","SupportedCurves":[29,23,30,25,24,256,257,258,259,260],"SupportedPoints":"AAEC","SignatureSchemes":[1027,1283,1539,2055,2056,2057,2058,2059,2052,2053,2054,1025,1281,1537,771,769,770,1026,1282,1538],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[772,771,770,769],"RemoteAddr":{"IP":"127.0.0.1","Port":37346,"Zone":""},"LocalAddr":{"IP":"127.0.0.1","Port":444,"Zone":""}}}}
{"level":"debug","ts":1703058541.5993276,"logger":"tls.handshake","msg":"no matching certificate; will choose from all certificates","identifier":"localhost"}
{"level":"debug","ts":1703058541.5993369,"logger":"tls.handshake","msg":"choosing certificate","identifier":"localhost","num_choices":1}
{"level":"debug","ts":1703058541.5997484,"logger":"tls.handshake","msg":"custom certificate selection results","identifier":"localhost","subjects":["ca1 client"],"managed":false,"issuer_key":"","hash":"698fb9489431bcc67ac926b90dfeb62613b774afacfcc03730c23044cea23c73"}
{"level":"debug","ts":1703058541.5997899,"logger":"tls.handshake","msg":"matched certificate in cache","remote_ip":"127.0.0.1","remote_port":"37346","subjects":["ca1 client"],"managed":false,"expiration":1705581914,"hash":"698fb9489431bcc67ac926b90dfeb62613b774afacfcc03730c23044cea23c73"}
```

I continued testing and my test setup above has the issue that both certs in the bundle have the same subject: `Subject: C = AU, ST = Some-State, O = Internet Widgits Pty Ltd` which is the OpenSSL default but not realistic.

So I modified the CA creation to include a unique subject:

```
for ca in ca1 ca2; do mkdir -p $ca; openssl genrsa -out $ca/$ca.key 2048; openssl req -x509 -batch -new -nodes -key $ca/$ca.key -sha256 -days 300 -subj "/CN=$ca root CA" -out $ca/$ca.crt; openssl genrsa -out $ca/$ca-mtls.key 4096; openssl req -new -key $ca/$ca-mtls.key -out $ca/$ca-mtls.csr -sha256 -subj "/CN=$ca client"; openssl x509 -req -days 300 -in $ca/$ca-mtls.csr -sha256 -CA $ca/$ca.crt -CAkey $ca/$ca.key -CAcreateserial -out $ca/$ca-mtls.crt -extfile mtls.conf; done

cat ca?/ca?.crt > ca_bundle.pem
```

Now when I repeat the test with both client certs:
```
for ca in ca1 ca2; do curl -k --cert $ca/$ca-mtls.crt --key $ca/$ca-mtls.key https://localhost:443/; echo; done
Bundle OK
curl: (56) OpenSSL SSL_read: OpenSSL/3.0.11: error:0A000418:SSL routines::tlsv1 alert unknown ca, errno 0
```
So with unique subjects in the bundle certs, the FIRST CA in the bundle actually works, but the second doesn't.

Logs... init config:
```javascript
{"level":"debug","ts":1703059660.0885599,"logger":"http.auto_https","msg":"adjusted config","tls":{"automation":{"policies":[{}]}},"http":{"servers":{"srv0":{"listen":[":443"],"routes":[{"handle":[{"body":"Bundle OK","handler":"static_response"}]}],"tls_connection_policies":[{"certificate_selection":{"any_tag":["cert0"]},"client_authentication":{"trusted_ca_certs":["MIIDDTCCAfWgAwIBAgIUPyJcwIkKCwjze2uZlx/QczsWNK0wDQYJKoZIhvcNAQELBQAwFjEUMBIGA1UEAwwLY2ExIHJvb3QgQ0EwHhcNMjMxMjIwMDgwNTMzWhcNMjQxMDE1MDgwNTMzWjAWMRQwEgYDVQQDDAtjYTEgcm9vdCBDQTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAMdw5wuGOOiMqEMZmNngEoK8YkLjjqXTEF9KicbLIeWlTnsJFxAosIW9GEg/xHtgWghilss3YVL3k7rkDngaWnu4cLOI9APXnx/nPrvlh1RFWuJeehrQ+lv00a9SRf7Ghl+vbqBOM7nkVioKMcjrC+CKc3dL2PJOYAiF9PLT9ay14H4KKKItbsXZ+03mtx+ZJoPswZmbA3E3w/T4GeYI16nblwaqbQhqZ5rCT81sgjrx5J/CT75Uydl0EX65li8eUx1TT7mCumIYS63K3NeK/fvSOedlBQvMAuI8MC350a7D4Mh1+f5kQMfXcnUCsRzwgIwLYiq59S31+LMlX3Wjar0CAwEAAaNTMFEwHQYDVR0OBBYEFDkv4GiGYdNlDisZf3fHRq7mmcMjMB8GA1UdIwQYMBaAFDkv4GiGYdNlDisZf3fHRq7mmcMjMA8GA1UdEwEB/wQFMAMBAf8wDQYJKoZIhvcNAQELBQADggEBAAWMpd7e3H+ruh/oD+APBuyMxToWYo/inGHVBFJrtHWIX3raaoFONXmmTTJ/qWrQeZn+wpsXPbDxeDUpV8MYg1cy2kEkG1mNPqgc2FeL6nKsfprz7WlCttyNIMYRN3U6BdpUh+AYOFpX6QNMHJZ5OvWlSKsb0x/+Kb5MHvUlCXEs+mQqJeGJ9eMAE6aecgfJ01qKEnuTWDKpIH5f/JH0zBE8F1V7uac+DI2d9sZohTrSUeZYSrTaziLvnvx8i9hF37K15H6Ddso4vYZa223mHgIBRiseJZT916sicI0h89Bz8bf5Wc21GE53CART8XAc570On2DcqFonXyK2HRq0qzg="],"mode":"require_and_verify"}}],"automatic_https":{"disable":true}}}}}
...
```
.. still has only one element in the trusted CA certs list.

OK request with client cert signed by first CA in bundle:
```javascript
{"level":"debug","ts":1703059688.3347592,"logger":"events","msg":"event","name":"tls_get_certificate","id":"c356153e-795f-4f73-8450-a309ac822b74","origin":"tls","data":{"client_hello":{"CipherSuites":[4866,4867,4865,49196,49200,159,52393,52392,52394,49195,49199,158,49188,49192,107,49187,49191,103,49162,49172,57,49161,49171,51,157,156,61,60,53,47,255],"ServerName":"localhost","SupportedCurves":[29,23,30,25,24,256,257,258,259,260],"SupportedPoints":"AAEC","SignatureSchemes":[1027,1283,1539,2055,2056,2057,2058,2059,2052,2053,2054,1025,1281,1537,771,769,770,1026,1282,1538],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[772,771,770,769],"RemoteAddr":{"IP":"127.0.0.1","Port":46818,"Zone":""},"LocalAddr":{"IP":"127.0.0.1","Port":443,"Zone":""}}}}
{"level":"debug","ts":1703059688.3350499,"logger":"tls.handshake","msg":"no matching certificate; will choose from all certificates","identifier":"localhost"}
{"level":"debug","ts":1703059688.3350613,"logger":"tls.handshake","msg":"choosing certificate","identifier":"localhost","num_choices":1}
{"level":"debug","ts":1703059688.33514,"logger":"tls.handshake","msg":"custom certificate selection results","identifier":"localhost","subjects":["ca1 client"],"managed":false,"issuer_key":"","hash":"55cabd733b3538617ac355c57ba154a6308612623863f17994eb8e478d6f34b9"}
{"level":"debug","ts":1703059688.335157,"logger":"tls.handshake","msg":"matched certificate in cache","remote_ip":"127.0.0.1","remote_port":"46818","subjects":["ca1 client"],"managed":false,"expiration":1728979538,"hash":"55cabd733b3538617ac355c57ba154a6308612623863f17994eb8e478d6f34b9"}
```

FAILed request with client cert signed by second CA in bundle:
```javascript
{"level":"debug","ts":1703059688.401022,"logger":"events","msg":"event","name":"tls_get_certificate","id":"ada267cc-b6eb-4ba4-b069-3ac9ab64b8ae","origin":"tls","data":{"client_hello":{"CipherSuites":[4866,4867,4865,49196,49200,159,52393,52392,52394,49195,49199,158,49188,49192,107,49187,49191,103,49162,49172,57,49161,49171,51,157,156,61,60,53,47,255],"ServerName":"localhost","SupportedCurves":[29,23,30,25,24,256,257,258,259,260],"SupportedPoints":"AAEC","SignatureSchemes":[1027,1283,1539,2055,2056,2057,2058,2059,2052,2053,2054,1025,1281,1537,771,769,770,1026,1282,1538],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[772,771,770,769],"RemoteAddr":{"IP":"127.0.0.1","Port":46826,"Zone":""},"LocalAddr":{"IP":"127.0.0.1","Port":443,"Zone":""}}}}
{"level":"debug","ts":1703059688.4011343,"logger":"tls.handshake","msg":"no matching certificate; will choose from all certificates","identifier":"localhost"}
{"level":"debug","ts":1703059688.4011443,"logger":"tls.handshake","msg":"choosing certificate","identifier":"localhost","num_choices":1}
{"level":"debug","ts":1703059688.4012086,"logger":"tls.handshake","msg":"custom certificate selection results","identifier":"localhost","subjects":["ca1 client"],"managed":false,"issuer_key":"","hash":"55cabd733b3538617ac355c57ba154a6308612623863f17994eb8e478d6f34b9"}
{"level":"debug","ts":1703059688.401226,"logger":"tls.handshake","msg":"matched certificate in cache","remote_ip":"127.0.0.1","remote_port":"46826","subjects":["ca1 client"],"managed":false,"expiration":1728979538,"hash":"55cabd733b3538617ac355c57ba154a6308612623863f17994eb8e478d6f34b9"}
{"level":"debug","ts":1703059688.4422126,"logger":"http.stdlib","msg":"http: TLS handshake error from 127.0.0.1:46826: tls: failed to verify certificate: x509: certificate signed by unknown authority"}
```

Summary:

- If a `trusted_ca_cert_file` contains multiple CA certs with different subjects (normal case), only the first one seems to work
- If a `trusted_ca_cert_file` contains multiple CA certs with the same subject, none of them work
- If each CA is listed in a separate `trusted_ca_cert_file` they all work, even if they have the same subject
Okay, I think I see the problem. PR incoming shortly.

Fix patch:
diff --git a/caddyconfig/httpcaddyfile/builtins.go b/caddyconfig/httpcaddyfile/builtins.go
index 94ca007d2f5..22b2f01fa25 100644
--- a/caddyconfig/httpcaddyfile/builtins.go
+++ b/caddyconfig/httpcaddyfile/builtins.go
@@ -246,16 +246,26 @@ func parseTLS(h Helper) ([]ConfigValue, error) {
 						if err != nil {
 							return nil, err
 						}
-						block, _ := pem.Decode(certDataPEM)
-						if block == nil || block.Type != "CERTIFICATE" {
-							return nil, h.Errf("no CERTIFICATE pem block found in %s", h.Val())
+						// while block is not nil, we have more certificates in the file
+						for block, rest := pem.Decode(certDataPEM); block != nil; block, rest = pem.Decode(rest) {
+							if block.Type != "CERTIFICATE" {
+								return nil, h.Errf("no CERTIFICATE pem block found in %s", filename)
+							}
+							if subdir == "trusted_ca_cert_file" {
+								cp.ClientAuthentication.TrustedCACerts = append(
+									cp.ClientAuthentication.TrustedCACerts,
+									base64.StdEncoding.EncodeToString(block.Bytes),
+								)
+							} else {
+								cp.ClientAuthentication.TrustedLeafCerts = append(
+									cp.ClientAuthentication.TrustedLeafCerts,
+									base64.StdEncoding.EncodeToString(block.Bytes),
+								)
+							}
 						}
-						if subdir == "trusted_ca_cert_file" {
-							cp.ClientAuthentication.TrustedCACerts = append(cp.ClientAuthentication.TrustedCACerts,
-								base64.StdEncoding.EncodeToString(block.Bytes))
-						} else {
-							cp.ClientAuthentication.TrustedLeafCerts = append(cp.ClientAuthentication.TrustedLeafCerts,
-								base64.StdEncoding.EncodeToString(block.Bytes))
+						// if we decoded nothing, return an error
+						if len(cp.ClientAuthentication.TrustedCACerts) == 0 && len(cp.ClientAuthentication.TrustedLeafCerts) == 0 {
+							return nil, h.Errf("no CERTIFICATE pem block found in %s", filename)
 						}
 
 					default:


Test patch: