Repository: Mailu/Mailu

Issue:
Root CA is not included in certificate chain for DANE validation
Thank you for opening an issue with Mailu. Please understand that issues are meant for bugs and enhancement-requests.
For **user-support questions**, reach out to us  on [matrix](https://matrix.to/#/#mailu:tedomum.net).

To be able to help you best, we need some more information.

## Before you open your issue
- [X] Check if no issue or pull-request for this already exists.
- [X] Check [documentation](https://mailu.io/master/) and [FAQ](https://mailu.io/master/faq.html). (Tip, use the search function on the documentation page)
- [X] You understand `Mailu` is made by volunteers in their **free time** — be conscise, civil and accept that delays can occur.
- [X] The title of the issue should be short and simple. It should contain specific terms related to the actual issue. Be specific while writing the title.

## Environment & Versions
### Environment
 - [X] docker-compose
 - [ ] kubernetes
 - [ ] docker swarm

### Versions
1.9

## Description
 I just upgraded to 1.9 and am trying to integrate DANE validation.  I got the following DANE Survey Notice:

"Your TLSA record designates a root CA hash, but, as is common, the root CA is not included in your certificate chain.  It would need to be included to work with DANE-TA(2), but simpler to use an intermediate CA hash instead."

## Replication Steps
I set a DNS TLSA record as recommended, with 2 1 1 designation.  I have DNSSEC activated for the relevant domain.  Certificates are issued through Let's Encrypt.

## Expected behaviour
Full validation of the certificate chain.

## Logs




Discussion:
Doh. Good catch
@vrjula would you mind trying with what's in [#2140](https://github.com/Mailu/Mailu/pull/2140/files) please?
I swapped my records to the generated "2 1 0" from yesterday's release and today I get the following message from the survey ([https://stats.dnssec-tools.org/](https://stats.dnssec-tools.org/)):

> Not all DANE implementations support "2 1 0" TLSA records
      with public keys of root CAs not included in the certificate
      chain. See <https://www.rfc-editor.org/rfc/rfc7671.html#section-5.2.2>
>
> For Let's Encrypt, publish the recommended "2 1 1" records instead:
      <https://dnssec-stats.ant.isi.edu/~viktor/x3hosts.html>
Grrr, what a mess.

If we pin the intermediaries we need 4 records and they would be valid for at most 3 years... I would rather we stick with what we have or consider more sensible alternatives (like creating a new CA locally -with a lifetime expressed in decades- that we would cross-sign with just for DANE enabled setups)
@csutcliff can you try with this please? If it works we could switch to that instead

```
_443._tcp.mail.example.com IN TLSA 2 0 0 3082056b30820353a0030201020211008210cfb0d240e3594463e0bb63828b00300d06092a864886f70d01010b0500304f310b300906035504061302555331293027060355040a1320496e7465726e65742053656375726974792052657365617263682047726f7570311530130603550403130c4953524720526f6f74205831301e170d3135303630343131303433385a170d3335303630343131303433385a304f310b300906035504061302555331293027060355040a1320496e7465726e65742053656375726974792052657365617263682047726f7570311530130603550403130c4953524720526f6f7420583130820222300d06092a864886f70d01010105000382020f003082020a0282020100ade82473f41437f39b9e2b57281c87bedcb7df38908c6e3ce657a078f775c2a2fef56a6ef6004f28dbde68866c4493b6b163fd14126bbf1fd2ea319b217ed1333cba48f5dd79dfb3b8ff12f1219a4bc18a8671694a66666c8f7e3c70bfad292206f3e4c0e680aee24b8fb7997e94039fd347977c99482353e838ae4f0a6f832ed149578c8074b6da2fd0388d7b0370211b75f2303cfa8faeddda63abeb164fc28e114b7ecf0be8ffb5772ef4b27b4ae04c12250c708d0329a0e15324ec13d9ee19bf10b34a8c3f89a36151deac870794f46371ec2ee26f5b9881e1895c34796c76ef3b906279e6dba49a2f26c5d010e10eded9108e16fbb7f7a8f7c7e50207988f360895e7e237960d36759efb0e72b11d9bbc03f94905d881dd05b42ad641e9ac0176950a0fd8dfd5bd121f352f28176cd298c1a80964776e4737baceac595e689d7f72d689c50641293e593edd26f524c911a75aa34c401f46a199b5a73a516e863b9e7d72a712057859ed3e5178150b038f8dd02f05b23e7b4a1c4b730512fcc6eae050137c439374b3ca74e78e1f0108d030d45b7136b407bac130305c48b7823b98a67d608aa2a32982ccbabd83041ba2830341a1d605f11bc2b6f0a87c863b46a8482a88dc769a76bf1f6aa53d198feb38f364dec82b0d0a28fff7dbe21542d422d0275de179fe18e77088ad4ee6d98b3ac6dd27516effbc64f533434f0203010001a3423040300e0603551d0f0101ff040403020106300f0603551d130101ff040530030101ff301d0603551d0e0416041479b459e67bb6e5e40173800888c81a58f6e99b6e300d06092a864886f70d01010b05000382020100551f58a9bcb2a850d00cb1d81a6920272908ac61755c8a6ef882e5692fd5f6564bb9b8731059d321977ee74c71fbb2d260ad39a80bea17215685f1500e59ebcee059e9bac915ef869d8f8480f6e4e99190dc179b621b45f06695d27c6fc2ea3bef1fcfcbd6ae27f1a9b0c8aefd7d7e9afa2204ebffd97fea912b22b1170e8ff28a345b58d8fc01c954b9b826cc8a8833894c2d843c82dfee965705ba2cbbf7c4b7c74e3b82be31c822737392d1c280a43939103323824c3c9f86b255981dbe29868c229b9ee26b3b573a82704ddc09c789cb0a074d6ce85d8ec9efceabc7bbb52b4e45d64ad026cce572ca086aa595e315a1f7a4edc92c5fa5fbffac28022ebed77bbbe3717b9016d3075e46537c3707428cd3c4969cd599b52ae0951a8048ae4c3907cecc47a452952bbab8fbadd233537de51d4d6dd5a1b1c7426fe64027355ca328b7078de78d3390e7239ffb509c796c46d5b415b3966e7e9b0c963ab8522d3fd65be1fb08c284fe24a8a389daac6ae1182ab1a843615bd31fdc3b8d76f22de88d75df17336c3d53fb7bcb415fffdca2d06138e196b8ac5d8b37d775d533c09911ae9d41c1727584be0241425f67244894d19b27be073fb9b84f817451e17ab7ed9d23e2bee0d52804133c31039edd7a6c8fc60718c67fde478e3f289e0406cfa5543477bdec899be91743df5bdb5ffe8e1e57a2cd409d7e6222dade1827
```
I'm afraid that I can't as it is rejected by my DNS provider (Cloudflare)

> The content length is too large. (Code: 81041)
After the first swap, my DANE validation is now passing:
![2022-01-06_09-37](https://user-images.githubusercontent.com/60139689/148409100-1391b3a4-36d9-4bcc-b8d0-50d7ebdd0bfa.png)

My favourite option has now become to cross-sign the LE certificate with a CA we would generate locally specifically for DANE. It would be small (ECDSA), could have a very long lifetime and could be used to cross-sign non-LE certs too

But since this is a much bigger piece of work, I'm not sure it would be wise to backport it to 1.9; we'll discuss it at the next meeting I guess. CC @Diman0 
> After the first swap, my DANE validation is now passing: ![2022-01-06_09-37](https://user-images.githubusercontent.com/60139689/148409100-1391b3a4-36d9-4bcc-b8d0-50d7ebdd0bfa.png)

Mine also passes that test with the "2 1 0" record but the dnssec-tools.org email still suggests that it will fail with some implementations
> My favourite option has now become to cross-sign the LE certificate with a CA we would generate locally specifically for DANE. It would be small (ECDSA), could have a very long lifetime and could be used to cross-sign non-LE certs too
> 
> But since this is a much bigger piece of work, I'm not sure it would be wise to backport it to 1.9; we'll discuss it at the next meeting I guess. CC @Diman0

That sounds like a good solution for a future version. In the mean time I've added the 4 records suggested for letsencrypt by https://dnssec-stats.ant.isi.edu/~viktor/x3hosts.html which should last a few years!
> Grrr, what a mess.
> 
> If we pin the intermediaries we need 4 records and they would be valid for at most 3 years... I would rather we stick with what we have or consider more sensible alternatives (like creating a new CA locally -with a lifetime expressed in decades- that we would cross-sign with just for DANE enabled setups)

Well, strictly speaking you don't need all four.  One is enough if you delay certificate rollover until the new CA hash has been published (along with the current) for a few TTLs.

This requires a more sophisticated rollover process, but that is needed in any case, since, as you note, the four hashes in question are not forever.

DANE requires monitoring, and robust automation that now and then needs to actually update the TLSA records, and wait for remote caches (and secondary servers) to catch up.

My advice would be to not use "2 X X" TLSA records at all, or at least not for low security "DV" CAs like Let's Encrypt. You're far better off with "3 1 1" (which is the approach used by <https://mailinabox.email>).

* The future key is generated, and a matching TLSA record is added to the RRset.
* A delay of a few TTLs/days expires
* A new certificate (perhaps from Let's Encrypt to satisfy non-DANE clients) is deployed with that key.
* The TLSA RR matching the old key is purged.

One way to handle this is to always generate the future key as soon as the new cert is deployed, that way you always have two TLSA RRs (current + next):

   https://mail.sys4.de/pipermail/dane-users/2018-February/000440.html

When publishing the next key, drop the now stale previous key at the same time.

Or perhaps four "3 1 1" records if deploying both an RSA and ECDSA cert (not common):

  https://mail.sys4.de/pipermail/dane-users/2017-August/000416.html
Finally, I very much DO NOT recommend the DANE TLSA *matching type* `Full(0)`, which places the entire certificate or public key in the TLSA record payload, at least not with RSA keys, which are quite large. This leads to rather hefty DNS UDP payloads, hindering interoperability and increasing response amplification.

Please stick with `SHA2-256(1)` hashes, and let the certs be conveyed over TCP in the TLS handshake.  Thus either "3 1 1" or "2 1 1", and avoid all the others.

Below are the survey stats for distinct TLSA RR parameters by base domain:
```
 count | usage | selector | mtype 
-------+-------+----------+-------
 16143 |     3 |        1 |     1
  1473 |     2 |        1 |     1
  1170 |     3 |        0 |     1
   415 |     3 |        1 |     2
   206 |     2 |        0 |     1
   176 |     3 |        0 |     2
    70 |     2 |        1 |     2
    24 |     2 |        0 |     2
    21 |     2 |        1 |     0       <-----
    13 |     1 |        1 |     1
    11 |     2 |        0 |     0
    10 |     3 |        1 |     0
     6 |     0 |        0 |     0
     5 |     0 |        1 |     1
     4 |     1 |        1 |     2
     4 |     0 |        1 |     2
     4 |     0 |        0 |     1
     4 |     1 |        0 |     1
     1 |     1 |        0 |     2
     1 |     0 |        0 |     2
     1 |     3 |        3 |     3
```
As you can see, "2 1 0" is not at all popular...
> This requires a more sophisticated rollover process, but that is needed in any case, since, as you note, the four hashes in question are not forever.
> 

That's why pinning the root CA makes sense; pinning something that expire in decades allows to ignore the rollover problem (for now).

> My advice would be to not use "2 X X" TLSA records at all, or at least not for low security "DV" CAs like Let's Encrypt. You're far better off with "3 1 1" (which is the approach used by https://mailinabox.email).
> 

Doing so would mean keeping the same key material and using larger key sizes... We could.
I've changed it from 4096bit RSA to 2048bit RSA to optimize handshake sizes... This obviously wouldn't be compatible with keeping the key material for decades.

> One way to handle this is to always generate the future key as soon as the new cert is deployed, that way you always have two TLSA RRs (current + next):
> 

Security-wise this doesn't make much sense; we are not in a scenario where the key material could be kept offline

> Or perhaps four "3 1 1" records if deploying both an RSA and ECDSA cert (not common):

We're already using two types of certs.
You should *not* be keeping any keys live for decades.  Since rollover should ultimately happen, far better to do it often and well, than to defer it to some unspecified future date.

Use "3 1 1" keys, rolled every 90 days or so, with prior TLSA updates. You can then use Let's Encrypt, without actually trusting it for DANE, since its domain control challenges are quite weak.

Yes, you can use a "2 1 1" RR matching a local CA (included in the server chain file and sent with the TLS handshake), but that only makes sense for large mail server clusters with a centrally managed PKI.  That is, for Google's `mx[1-4].smtp.goog`, but not for SOHO/vanity domain mail servers.
Thank you for taking the time to reply.

We've decided not long ago that maintaining an authoritative DNS server on behalf of our users was out of scope for the project... and so is programmatically managing their DNS records (we were looking at https://github.com/AnalogJ/lexicon/).

Based on the above, I see the following solutions:
1) ~~stick to 210 pinned to ISRG X1 and assume the fact that we don't go for the popular choice.~~ This isn't an option, see below
2) stop encouraging our users to deploy DANE/TLSA records; those that want them can of course still do it but at least we won't be advocating for "known problematic/exotic" configuration choices
3) use 311 records, stop changing the key material on certificate re-issuance; The rationale being that the security margin offered by a 4096 bit RSA certificate is likely good enough for the next decade (https://www.keylength.com/en/3/). We could warn about certificate age and let the user manage the rollover on his own time
4) use 211 records pinned to the intermediaries (E1 E2 R3 R4 today); We know when they are due to expire so we could display a big fat red warning in advance of their expiry... but it will be a recurring headache with a 3y periodicity at best.
5) use 211 records pinned to ISRG X1 (revert to the state before this ticket), and send ISRG X1 in the handshake (4096 extra bits! we were not doing that but we could accept that cost).
6) use 211 matching a local CA that we would send in the handshake (manage our own local PKI). The advantages over the previous solution being: 1) we have agility in terms of which CA is used and we could support user-managed certificates (and make DANE available to more users) 2) the CA would likely be using ECC and its certificate would be smaller than ISRG X1
This issue is NOT resolved by #2140, and needs to be reopened.  The change in https://github.com/Mailu/Mailu/blob/master/core/admin/mailu/models.py#L273-L279
puts the root CA's *key* (not full certificate) in the TLSA record.

Use of bare "2 1 0" keys (without the corresponding cert in the chain, making the full key in DNS redundant, and "2 1 1" a better choice) violates [RFC7671 Section 5.2.3](https://datatracker.ietf.org/doc/html/rfc7671#section-5.2.3):
```
   Thus, servers cannot rely on "DANE-TA(2) SPKI(1) Full(0)" TLSA
   records to be sufficient to authenticate chains issued by the
   associated public key in the absence of a corresponding certificate
   in the server's TLS certificate message.  Servers employing "2 1 0"
   TLSA records MUST include the corresponding TA certificate in their
   certificate chain.
```


> We've decided not long ago that maintaining an authoritative DNS server on behalf of our users was out of scope for the project... and so is programmatically managing their DNS records (we were looking at https://github.com/AnalogJ/lexicon/).

So, you've decided to not attempt to automate DNS TLSA record updates, with DANE supported only via long-term static data... [ If this is wrong, please clarify ]

> Based on the above, I see the following solutions:
> 
> 1. stick to 210 pinned to ISRG X1 and assume the fact that we don't go for the popular choice.

Popularity isn't the problem, it is *interoperability*, I don't expect Microsoft's or Cisco's (or pick some other random implementation that isn't mine, i.e. Postfix, Exim or OpenSSL) DANE verification stack to support bare "2 1 0" keys in the absence of a matching cert in the chain.  IIRC I've seen DANE verification code in "Go" that would fail to handle this...

My survey code also chooses to not support "2 1 0" except via a matching chain cert, and users with "2 1 0" TLSA records get nagging email from the survey...

> 2. stop encouraging our users to deploy DANE/TLSA records; those that want them can of course still do it but at least we won't be advocating for "known problematic/exotic" configuration choices

That's an option of course, though ideally I'd like to see turnkey solutions that support DANE, which is often better than users doing their own DANE integration, not always in a robust manner...

> 3. use 311 records, stop changing the key material on certificate re-issuance; The rationale being that the security margin offered by an a 4096 bit RSA certificate is likely good enough for the next decade (https://www.keylength.com/en/3/)

This is certainly a popular choice.  And you don't need 4096-bit RSA keys to get there, 2048-bits is plenty resistant to brute force attacks (even 1536 bits is well outside any plausible classical attack).  Indeed with the DNS root zone signed with a long-term 2048-bit RSA key, using 4096-bit RSA keys lower down the stack is rather futile.

> 4. use 211 records pinned to the intermediaries (E1 E2 R3 R4 today); We know when they are due to expire so we could display a big fat red warning in advance of their expiry... but it will be a recurring headache with a 3y periodicity at best.

That is plausible, but there's no good reason to trust LE issuers, this just reduces security.  That does not mean that one should not deploy LE certs to support non-DANE relying parties (or even, not recommended, MTA-STS).

> 5. use 211 records pinned to ISRG X1 (revert to the state before this ticket), and send ISRG X1 in the handshake (4096 extra bits! we were not doing that but we could accept that cost).

SMTP is not latency sensitive, there's no interactive user loading a webpage. There's little reason to avoid sending the root cert, at least on port 25.  You might have a case for port 465 or 587, with handheld devices sending email, but these could use a separate config sans root cert (those MUAs don't presently do DANE).

Thus, sending the X1 root is fine.  (Note that for submission, if you want to support old Android devices, what gets sent is a cross-cert for ISRG, signed by an expired DST root, but Android ignores root CA expiration).

> 6. use 211 matching a local CA that we would send in the handshake (manage our own local PKI). The advantages over the previous solution being: 1) we have agility in terms of which CA is used and we could support user-managed certificates (and make DANE available to more users) 2) the CA would likely be using ECC and its certificate smaller than ISRG X1

This is also fine, but not clear why it is better than "3 1 1", unless the same CA signs lots of keys, is kept off-line or in an HSM, is resistant to misissuance, ...

A "3 1 1" is basically CA + server key all in one, and compatible with actually using an LE cert (for a key that is stable while the TLSA RRset is unchanged).


I should perhaps mention that 3 years is not necessarily the upper bound on the validity of the key hashes for R3/E1/R4/E2.  The intermediate issuer certificates may well (this has happened previously) be renewed with the same underlying key, just new expiration dates, and various other metadata.  Time will tell...

Resolving patch:
diff --git a/core/admin/mailu/models.py b/core/admin/mailu/models.py
--- a/core/admin/mailu/models.py
+++ b/core/admin/mailu/models.py
@@ -276,7 +276,7 @@ def dns_tlsa(self):
         hostname = app.config['HOSTNAME']
         if app.config['TLS_FLAVOR'] in ('letsencrypt', 'mail-letsencrypt'):
             # current ISRG Root X1 (RSA 4096, O = Internet Security Research Group, CN = ISRG Root X1) @20210902
-            return f'_25._tcp.{hostname}. 86400 IN TLSA 2 1 0 30820222300d06092a864886f70d01010105000382020f003082020a0282020100ade82473f41437f39b9e2b57281c87bedcb7df38908c6e3ce657a078f775c2a2fef56a6ef6004f28dbde68866c4493b6b163fd14126bbf1fd2ea319b217ed1333cba48f5dd79dfb3b8ff12f1219a4bc18a8671694a66666c8f7e3c70bfad292206f3e4c0e680aee24b8fb7997e94039fd347977c99482353e838ae4f0a6f832ed149578c8074b6da2fd0388d7b0370211b75f2303cfa8faeddda63abeb164fc28e114b7ecf0be8ffb5772ef4b27b4ae04c12250c708d0329a0e15324ec13d9ee19bf10b34a8c3f89a36151deac870794f46371ec2ee26f5b9881e1895c34796c76ef3b906279e6dba49a2f26c5d010e10eded9108e16fbb7f7a8f7c7e50207988f360895e7e237960d36759efb0e72b11d9bbc03f94905d881dd05b42ad641e9ac0176950a0fd8dfd5bd121f352f28176cd298c1a80964776e4737baceac595e689d7f72d689c50641293e593edd26f524c911a75aa34c401f46a199b5a73a516e863b9e7d72a712057859ed3e5178150b038f8dd02f05b23e7b4a1c4b730512fcc6eae050137c439374b3ca74e78e1f0108d030d45b7136b407bac130305c48b7823b98a67d608aa2a32982ccbabd83041ba2830341a1d605f11bc2b6f0a87c863b46a8482a88dc769a76bf1f6aa53d198feb38f364dec82b0d0a28fff7dbe21542d422d0275de179fe18e77088ad4ee6d98b3ac6dd27516effbc64f533434f0203010001'
+            return f'_25._tcp.{hostname}. 86400 IN TLSA 2 1 1 0b9fa5a59eed715c26c1020c711b4f6ec42d58b0015e14337a39dad301c5afc3'
 
     @property
     def dkim_key(self):
diff --git a/core/nginx/config.py b/core/nginx/config.py
--- a/core/nginx/config.py
+++ b/core/nginx/config.py
@@ -34,6 +34,22 @@
     "notls": None
 }[args["TLS_FLAVOR"]]
 
+def format_for_nginx(fullchain, output):
+    """ We may want to strip ISRG Root X1 out """
+    if not os.path.exists(fullchain):
+        return
+    split = '-----END CERTIFICATE-----\n'
+    with open(fullchain, 'r') as pem:
+        certs = [f'{cert}{split}' for cert in pem.read().split(split) if cert]
+    if len(certs)>2 and os.getenv('LETSENCRYPT_SHORTCHAIN'):
+        del certs[-1]
+    with open(output, 'w') as pem:
+        pem.write(''.join(certs))
+
+if args['TLS_FLAVOR'] in ['letsencrypt', 'mail-letsencrypt']:
+    format_for_nginx('/certs/letsencrypt/live/mailu/fullchain.pem', '/certs/letsencrypt/live/mailu/nginx-chain.pem')
+    format_for_nginx('/certs/letsencrypt/live/mailu-ecdsa/fullchain.pem', '/certs/letsencrypt/live/mailu-ecdsa/nginx-chain.pem')
+
 if args["TLS"] and not all(os.path.exists(file_path) for file_path in args["TLS"]):
     print("Missing cert or key file, disabling TLS")
     args["TLS_ERROR"] = "yes"
diff --git a/core/nginx/letsencrypt.py b/core/nginx/letsencrypt.py
--- a/core/nginx/letsencrypt.py
+++ b/core/nginx/letsencrypt.py
@@ -32,28 +32,11 @@
     "--post-hook", "/config.py"
 ]
 
-def format_for_nginx(fullchain, output):
-    """ We may want to strip ISRG Root X1 out
-    """
-    certs = []
-    with open(fullchain, 'r') as pem:
-        cert = ''
-        for line in pem:
-            cert += line
-            if '-----END CERTIFICATE-----' in line:
-                certs += [cert]
-                cert = ''
-    with open(output, 'w') as pem:
-        for cert in certs[:-1] if len(certs)>2 and os.getenv('LETSENCRYPT_SHORTCHAIN', default="False") else certs:
-            pem.write(cert)
-
 # Wait for nginx to start
 time.sleep(5)
 
 # Run certbot every day
 while True:
     subprocess.call(command)
-    format_for_nginx('/certs/letsencrypt/live/mailu/fullchain.pem', '/certs/letsencrypt/live/mailu/nginx-chain.pem')
     subprocess.call(command2)
-    format_for_nginx('/certs/letsencrypt/live/mailu-ecdsa/fullchain.pem', '/certs/letsencrypt/live/mailu-ecdsa/nginx-chain.pem')
     time.sleep(86400)


Test patch: