Repository: Mailu/Mailu

Issue:
helm/k8s deployments seem to have problems with DNS resolution;  Mailu expects validated dnssec records
## Description
As the default k8s dns setup is not working correctly with dnssec, we will (like the docker setup) use unbound as resolver, but to allow us to continue using the internal k8s-dns system, we need a dns forwarder setting.

## Replication Steps
Setup mailu with mailu helm chart in k8s

I will provide a suitable fix myself.


Discussion:
This is related to #2163

We have talked about it on #mailu-dev recently. My preferred option would be to do the opposite: configure k8s' resolver to use unbound (that would run as a service in a pod) as upstream DNS resolver. The pro of doing it that way is to ensure cluster stability if the unbound pod is having problems...

https://kubernetes.io/docs/tasks/administer-cluster/dns-custom-nameservers/#coredns-configuration-equivalent-to-kube-dns explains how to configure/change the upstream for the two common cases (coredns or kube-dns). Maybe all we need is some documentation in the FAQ?
@devfaz which DNS resolver is your k8s deployment configured to use? what is its upstream?

Can you run ``dig @<ip> +adflag example.org`` on the various resolvers and track down where you are loosing the AD flag please?
@nextgens did some tests, the adflag got lost asking the dns on my router (openwrt / dnsmasq). I replaced the upstream with google-dns and now im getting the adflag also in k8s.

So, maybe we should change the WARN-msg in admin-pod to include an link to some docs about "how to change your k8s-dns-server to one allowing successfully do dnssec"
Yes, I do believe that it's a documentation problem more than anything else.

For dnsmasq you can make it perform the validation https://wiki.gentoo.org/wiki/Dnsmasq/en#DNSSEC or configure it to pass the validation of upstream servers through.

I'd accept and backport a PR that adds a FAQ entry in the documentation and changes the message to point to it (https://mailu.io/1.9/faq.html#technical-issues)
Thanks for the link, but in openwrt its just a tick in a checkbox ;)

I will try to provide the requested PR.
@nextgens so after undoing my unbound change on my cluster, im getting again the admin-pod-crit:
`
CRITICAL:root:Your DNS resolver at 192.168.102.10 isn't doing DNSSEC validation; Please use another resolver or enable unbound via https://setup.mailu.io.
`

Background infos:
- 192.168.101.1 => dnsmasq on openwrt (dnssec enabled)
- 192.168.102.10 => kube-dns svc ip
- commands executed on debian-pod running in the same namespace.

`
# ask kube-dns for ad-flag => bad
root@bash:/# dig @192.168.102.10 +adflag example.org

; <<>> DiG 9.16.22-Debian <<>> @192.168.102.10 +adflag example.org
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 28083
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
; COOKIE: ac945c2f55e83efc (echoed)
;; QUESTION SECTION:
;example.org.			IN	A

;; ANSWER SECTION:
example.org.		27	IN	A	93.184.216.34

;; Query time: 4 msec
;; SERVER: 192.168.102.10#53(192.168.102.10)
;; WHEN: Tue Feb 15 07:50:49 UTC 2022
;; MSG SIZE  rcvd: 79

# ask openwrt-dns for ad-flag => good
root@bash:/# dig @192.168.101.1 +adflag example.org

; <<>> DiG 9.16.22-Debian <<>> @192.168.101.1 +adflag example.org
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 57895
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;example.org.			IN	A

;; ANSWER SECTION:
example.org.		17625	IN	A	93.184.216.34

;; Query time: 4 msec
;; SERVER: 192.168.101.1#53(192.168.101.1)
;; WHEN: Tue Feb 15 07:50:58 UTC 2022
;; MSG SIZE  rcvd: 56

# ask kube-dns for dnssec => good.
root@bash:/# dig @192.168.102.10 +dnssec example.org

; <<>> DiG 9.16.22-Debian <<>> @192.168.102.10 +dnssec example.org
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 10926
;; flags: qr aa rd ra ad; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 4096
; COOKIE: 1388011d98e57af4 (echoed)
;; QUESTION SECTION:
;example.org.			IN	A

;; ANSWER SECTION:
example.org.		6	IN	RRSIG	A 8 2 86400 20220222143712 20220201175836 14665 example.org. gVEFxer1XAPl/NbmyRRDWI3eOge/UVXVZ5BN4/62HDwn1NVAtZybjy70 T42FN6nvt9ARWwTWZBs+DF7+ohE66NQNiKYefXapMNdbJS4N6NbpiSn9 ZlkdUQzR/JR2onNvHeUclIYVL9spYJszECTPV5erz7H1g9+k71JJX+7O WFc=
example.org.		6	IN	A	93.184.216.34

;; Query time: 0 msec
;; SERVER: 192.168.102.10#53(192.168.102.10)
;; WHEN: Tue Feb 15 07:51:10 UTC 2022
;; MSG SIZE  rcvd: 261


# ask openwrt-dns for dnssec-flag => good
root@bash:/# dig @192.168.101.1 +dnssec example.org

; <<>> DiG 9.16.22-Debian <<>> @192.168.101.1 +dnssec example.org
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 49257
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 512
;; QUESTION SECTION:
;example.org.			IN	A

;; ANSWER SECTION:
example.org.		17457	IN	A	93.184.216.34
example.org.		17457	IN	RRSIG	A 8 2 86400 20220222143712 20220201175836 14665 example.org. gVEFxer1XAPl/NbmyRRDWI3eOge/UVXVZ5BN4/62HDwn1NVAtZybjy70 T42FN6nvt9ARWwTWZBs+DF7+ohE66NQNiKYefXapMNdbJS4N6NbpiSn9 ZlkdUQzR/JR2onNvHeUclIYVL9spYJszECTPV5erz7H1g9+k71JJX+7O WFc=

;; Query time: 4 msec
;; SERVER: 192.168.101.1#53(192.168.101.1)
;; WHEN: Tue Feb 15 07:53:46 UTC 2022
;; MSG SIZE  rcvd: 227
`

so seems CoreDNS is still having issues with the ad-flag, but works if you request +DNSSEC. That just FYI - i will try to investigate later.
Seems CoreDNS changed AD-flag behavior in 1.8.5 (im using k8s 1.23.3 which includes CoreDNS 1.8.6): https://coredns.io/2021/09/10/coredns-1.8.5-release/ => https://github.com/coredns/coredns/pull/4736.
I've replied on that PR (probably the wrong place tbh).

coredns ignores https://datatracker.ietf.org/doc/html/rfc6840#section-5.8 ... which makes perfectly clear that DO or AD in the query should be treated the same. This is a coredns bug.
> I've replied on that PR (probably the wrong place tbh).
> 
> coredns ignores https://datatracker.ietf.org/doc/html/rfc6840#section-5.8 ... which makes perfectly clear that DO or AD in the query should be treated the same. This is a coredns bug.

Yes, you are right - but how can we ensure mailu is working on k8s even with this bug?

I see 2 ways:

1. Apply the workaround to the start.py and hope the real tools are also setting the DO-flag.
2. Give users an easy way to workaround coreDNS on k8s completly till the bug is fixed. (unbound resolver ;) )
> Yes, you are right - but how can we ensure mailu is working on k8s even with this bug?

You can test it out with something like https://havedane.net/ or try to find out what postfix actually does (share a packet capture)

@micw do you want to shime in?
FYI: seems upstream (coreDNS) is fixing this quickly, so im just doing nothing and waiting for a new coreDNS version.

Resolving patch:
diff --git a/core/admin/mailu/utils.py b/core/admin/mailu/utils.py
--- a/core/admin/mailu/utils.py
+++ b/core/admin/mailu/utils.py
@@ -47,7 +47,7 @@ def handle_needs_login():
 
 # DNS stub configured to do DNSSEC enabled queries
 resolver = dns.resolver.Resolver()
-resolver.use_edns(0, 0, 1232)
+resolver.use_edns(0, dns.flags.DO, 1232)
 resolver.flags = dns.flags.AD | dns.flags.RD
 
 def has_dane_record(domain, timeout=10):
@@ -56,7 +56,6 @@ def has_dane_record(domain, timeout=10):
         if result.response.flags & dns.flags.AD:
             for record in result:
                 if isinstance(record, dns.rdtypes.ANY.TLSA.TLSA):
-                    record.validate()
                     if record.usage in [2,3] and record.selector in [0,1] and record.mtype in [0,1,2]:
                         return True
     except dns.resolver.NoNameservers:
diff --git a/core/admin/start.py b/core/admin/start.py
--- a/core/admin/start.py
+++ b/core/admin/start.py
@@ -28,7 +28,7 @@ def test_DNS():
     import time
     # DNS stub configured to do DNSSEC enabled queries
     resolver = dns.resolver.Resolver()
-    resolver.use_edns(0, 0, 1232)
+    resolver.use_edns(0, dns.flags.DO, 1232)
     resolver.flags = dns.flags.AD | dns.flags.RD
     nameservers = resolver.nameservers
     for ns in nameservers:
@@ -37,11 +37,11 @@ def test_DNS():
             try:
                 result = resolver.resolve('example.org', dns.rdatatype.A, dns.rdataclass.IN, lifetime=10)
             except Exception as e:
-                log.critical("Your DNS resolver at %s is not working (%s). Please use another resolver or enable unbound via https://setup.mailu.io.", ns, e);
+                log.critical("Your DNS resolver at %s is not working (%s). Please see https://mailu.io/master/faq.html#the-admin-container-won-t-start-and-its-log-says-critical-your-dns-resolver-isn-t-doing-dnssec-validation", ns, e);
             else:
                 if result.response.flags & dns.flags.AD:
                     break
-                log.critical("Your DNS resolver at %s isn't doing DNSSEC validation; Please use another resolver or enable unbound via https://setup.mailu.io.", ns)
+                log.critical("Your DNS resolver at %s isn't doing DNSSEC validation; Please see https://mailu.io/master/faq.html#the-admin-container-won-t-start-and-its-log-says-critical-your-dns-resolver-isn-t-doing-dnssec-validation.", ns)
             time.sleep(5)
 
 test_DNS()


Test patch: