Repository: Mailu/Mailu

Issue:
Mail Rejected/Failed to Send to Relay HOST - DANE exceptions in ADMIN, Similar in POSTFIX
Thank you for opening an issue with Mailu. Please understand that issues are meant for bugs and enhancement-requests.
For **user-support questions**, reach out to us  on [matrix](https://matrix.to/#/#mailu:tedomum.net).

To be able to help you best, we need some more information.

## Before you open your issue
- [ X] Check if no issue or pull-request for this already exists.
- [ ] Check [documentation](https://mailu.io/master/) and [FAQ](https://mailu.io/master/faq.html). (Tip, use the search function on the documentation page)
- [ ] You understand `Mailu` is made by volunteers in their **free time** — be conscise, civil and accept that delays can occur.
- [ ] The title of the issue should be short and simple. It should contain specific terms related to the actual issue. Be specific while writing the title.

## Environment & Versions
### Environment
 - [ ] docker-compose
 - [ X] kubernetes
 - [ ] docker swarm

### Versions
To find your version, get the image name of a mailu container and read  the version from the tag (example for version 1.7).
```
$> docker ps -a | grep mailu
140b09d4b09c    mailu/roundcube:1.7    "docker-php-entrypoi…"    2 weeks ago    Up 2 days (healthy)    80/tcp
$> grep MAILU_VERSION docker-compose.yml mailu.env
```

LATEST VERSION (from helm chart)

## Description
Further explain the bug in a few words. It should be clear what the unexpected behaviour is.  Share it in an easy-to-understand language.

We are sending mail to mailu, this was working with our configuration for about 1 year so far, during a recent K8s upgrade and following, we see that pod were redployed and now we see the errors in the logs when we are doing the same STARTTLS submit to the mail server that we have been doing prior

## Replication Steps
No idea, update the mailu to the latest release and that is what we saw

## Expected behaviour
the mail to work

## Logs
Here is the log from the ADMIN component, you can see its throwing an exception, we did the dig and nsloookup on the pod and there is no issues resoving the host, we did NETCAT and can connect from the ADMIN pod and the POSTFIX PODS and can do submit over netcat (command line ELHO/HELO, etc)

ADMIN LOG 


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/flask/app.py", line 2292, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/lib/python3.9/site-packages/flask/app.py", line 1815, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/lib/python3.9/site-packages/flask/app.py", line 1718, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/lib/python3.9/site-packages/flask/_compat.py", line 35, in reraise
    raise value
  File "/usr/lib/python3.9/site-packages/flask/app.py", line 1813, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/lib/python3.9/site-packages/flask/app.py", line 1799, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/app/mailu/internal/views/postfix.py", line 12, in postfix_dane_map
    return flask.jsonify('dane-only') if utils.has_dane_record(domain_name) else flask.abort(404)
  File "/app/mailu/utils.py", line 61, in has_dane_record
    return app.config['DEFER_ON_TLS_ERROR']
NameError: name 'app' is not defined
[2021-09-23 23:38:17,764] WARNING in utils: Unable to lookup the TLSA record for cerence-cloud.mail.protection.outlook.com. Is the DNSSEC zone okay on https://dnsviz.net/d/-cloud.mail.protection.outlook.com/dnssec/?
[2021-09-23 23:38:17,765] ERROR in app: Exception on /internal/postfix/dane/-cloud.mail.protection.outlook.com [GET]
Traceback (most recent call last):
  File "/app/mailu/utils.py", line 49, in has_dane_record
    result = resolver.query(f'_25._tcp.{domain}', dns.rdatatype.TLSA,dns.rdataclass.IN, lifetime=timeout)
  File "/usr/lib/python3.9/site-packages/dns/resolver.py", line 898, in query
    raise NoNameservers(request=request, errors=errors)
dns.resolver.NoNameservers: All nameservers failed to answer the query _25._tcp.cloud.mail.protection.outlook.com. IN TLSA: Server 172.18.0.2 UDP port 53 answered SERVFAIL

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/flask/app.py", line 2292, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/lib/python3.9/site-packages/flask/app.py", line 1815, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/lib/python3.9/site-packages/flask/app.py", line 1718, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/lib/python3.9/site-packages/flask/_compat.py", line 35, in reraise
    raise value
  File "/usr/lib/python3.9/site-packages/flask/app.py", line 1813, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/lib/python3.9/site-packages/flask/app.py", line 1799, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/app/mailu/internal/views/postfix.py", line 12, in postfix_dane_map
    return flask.jsonify('dane-only') if utils.has_dane_record(domain_name) else flask.abort(404)
  File "/app/mailu/utils.py", line 61, in has_dane_record
    return app.config['DEFER_ON_TLS_ERROR']
NameError: name 'app' is not defined



FROM THE POSTFIX LOG (when we are doing a submit of a mail with the same client we have used since we started the development / use of the service)



Sep 23 23:45:27 mail postfix/smtpd[2583]: connect from 10-196-225-85.secops-mail-service-mailu-front.secops-system.svc.cluster.local[10.196.225.85]
Sep 23 23:45:27 mail postfix/smtpd[2583]: discarding EHLO keywords: PIPELINING
Sep 23 23:45:27 mail postfix/smtpd[2583]: discarding EHLO keywords: PIPELINING
Sep 23 23:45:28 mail postfix/smtpd[2583]: 516185A0C77: client=10-196-226-1.internal-istio-ingressgateway.istio-system.svc.cluster.local[10.196.226.1], sasl_method=XCLIENT, sasl_username=demoadm@mail.
Sep 23 23:45:28 mail postfix/cleanup[2719]: 516185A0C77: replace: header Received: from localhost (10-196-226-1.internal-istio-ingressgateway.istio-system.svc.cluster.local [10.196.226.1])??by mail. (Postfix) with ESMTPA id 516185A0C77??for <lmc from 10-196-226-1.internal-istio-ingressgateway.istio-system.svc.cluster.local[10.196.226.1]; from=<demoadm@systems> to=<lmcdasm@gmail.com> proto=ESMTP helo=<localhost>: Received: from authenticated-user (mail.[10.196.225.152])??by mail. (Postfix) with ESMTPA id 516185A0C77??for <lmcdasm@gmail.com>; Thu, 23 Sep 2021 23:45:28 +0000 (UTC)
Sep 23 23:45:28 mail postfix/cleanup[2719]: 516185A0C77: message-id=<>
Sep 23 23:45:28 mail postfix/qmgr[377]: 516185A0C77: from=<demoadm@.systems>, size=380, nrcpt=1 (queue active)
Sep 23 23:45:28 mail postfix/smtpd[2583]: disconnect from 10-196-226-1.internal-istio-ingressgateway.istio-system.svc.cluster.local[10.196.226.1] ehlo=2 xclient=0/1 mail=1 rcpt=1 data=1 quit=1 commands=6/7
ERROR:root:Error when processing request
Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/podop/postfix.py", line 104, in process_request
    result = await table.get(key)
  File "/usr/lib/python3.9/site-packages/podop/table.py", line 35, in get
    raise Exception(request.status)
Exception: 500
Sep 23 23:45:28 mail postfix/smtp[2720]: warning: socketmap:unix:/tmp/podop.socket:dane socketmap server temporary error: unknown error
Sep 23 23:45:28 mail postfix/smtp[2720]: warning: socketmap:unix:/tmp/podop.socket:dane lookup error for "-cloud.mail.protection.outlook.com"
Sep 23 23:45:28 mail postfix/smtp[2720]: warning: smtp_tls_policy_maps, next-hop destination "-cloud.mail.protection.outlook.com": policy table lookup error
Sep 23 23:45:28 mail postfix/smtp[2720]: warning: TLS policy lookup for -cloud.mail.protection.outlook.com/-cloud.mail.protection.outlook.com: client TLS configuration problem
Sep 23 23:45:28 mail postfix/smtp[2720]: warning: TLS policy lookup for -cloud.mail.protection.outlook.com/-cloud.mail.protection.outlook.com: client TLS configuration problem
Sep 23 23:45:28 mail postfix/smtp[2720]: 516185A0C77: to=<lmcdasm@gmail.com>, relay=none, delay=0.8, delays=0.36/0.1/0.34/0, dsn=4.7.5, status=deferred (client TLS configuration problem)




Mail Rejected/Failed to Send to Relay HOST - DANE exceptions in ADMIN, Similar in POSTFIX
Thank you for opening an issue with Mailu. Please understand that issues are meant for bugs and enhancement-requests.
For **user-support questions**, reach out to us  on [matrix](https://matrix.to/#/#mailu:tedomum.net).

To be able to help you best, we need some more information.

## Before you open your issue
- [ X] Check if no issue or pull-request for this already exists.
- [ ] Check [documentation](https://mailu.io/master/) and [FAQ](https://mailu.io/master/faq.html). (Tip, use the search function on the documentation page)
- [ ] You understand `Mailu` is made by volunteers in their **free time** — be conscise, civil and accept that delays can occur.
- [ ] The title of the issue should be short and simple. It should contain specific terms related to the actual issue. Be specific while writing the title.

## Environment & Versions
### Environment
 - [ ] docker-compose
 - [ X] kubernetes
 - [ ] docker swarm

### Versions
To find your version, get the image name of a mailu container and read  the version from the tag (example for version 1.7).
```
$> docker ps -a | grep mailu
140b09d4b09c    mailu/roundcube:1.7    "docker-php-entrypoi…"    2 weeks ago    Up 2 days (healthy)    80/tcp
$> grep MAILU_VERSION docker-compose.yml mailu.env
```

LATEST VERSION (from helm chart)

## Description
Further explain the bug in a few words. It should be clear what the unexpected behaviour is.  Share it in an easy-to-understand language.

We are sending mail to mailu, this was working with our configuration for about 1 year so far, during a recent K8s upgrade and following, we see that pod were redployed and now we see the errors in the logs when we are doing the same STARTTLS submit to the mail server that we have been doing prior

## Replication Steps
No idea, update the mailu to the latest release and that is what we saw

## Expected behaviour
the mail to work

## Logs
Here is the log from the ADMIN component, you can see its throwing an exception, we did the dig and nsloookup on the pod and there is no issues resoving the host, we did NETCAT and can connect from the ADMIN pod and the POSTFIX PODS and can do submit over netcat (command line ELHO/HELO, etc)

ADMIN LOG 


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/flask/app.py", line 2292, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/lib/python3.9/site-packages/flask/app.py", line 1815, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/lib/python3.9/site-packages/flask/app.py", line 1718, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/lib/python3.9/site-packages/flask/_compat.py", line 35, in reraise
    raise value
  File "/usr/lib/python3.9/site-packages/flask/app.py", line 1813, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/lib/python3.9/site-packages/flask/app.py", line 1799, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/app/mailu/internal/views/postfix.py", line 12, in postfix_dane_map
    return flask.jsonify('dane-only') if utils.has_dane_record(domain_name) else flask.abort(404)
  File "/app/mailu/utils.py", line 61, in has_dane_record
    return app.config['DEFER_ON_TLS_ERROR']
NameError: name 'app' is not defined
[2021-09-23 23:38:17,764] WARNING in utils: Unable to lookup the TLSA record for cerence-cloud.mail.protection.outlook.com. Is the DNSSEC zone okay on https://dnsviz.net/d/-cloud.mail.protection.outlook.com/dnssec/?
[2021-09-23 23:38:17,765] ERROR in app: Exception on /internal/postfix/dane/-cloud.mail.protection.outlook.com [GET]
Traceback (most recent call last):
  File "/app/mailu/utils.py", line 49, in has_dane_record
    result = resolver.query(f'_25._tcp.{domain}', dns.rdatatype.TLSA,dns.rdataclass.IN, lifetime=timeout)
  File "/usr/lib/python3.9/site-packages/dns/resolver.py", line 898, in query
    raise NoNameservers(request=request, errors=errors)
dns.resolver.NoNameservers: All nameservers failed to answer the query _25._tcp.cloud.mail.protection.outlook.com. IN TLSA: Server 172.18.0.2 UDP port 53 answered SERVFAIL

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/flask/app.py", line 2292, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/lib/python3.9/site-packages/flask/app.py", line 1815, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/lib/python3.9/site-packages/flask/app.py", line 1718, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/lib/python3.9/site-packages/flask/_compat.py", line 35, in reraise
    raise value
  File "/usr/lib/python3.9/site-packages/flask/app.py", line 1813, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/lib/python3.9/site-packages/flask/app.py", line 1799, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/app/mailu/internal/views/postfix.py", line 12, in postfix_dane_map
    return flask.jsonify('dane-only') if utils.has_dane_record(domain_name) else flask.abort(404)
  File "/app/mailu/utils.py", line 61, in has_dane_record
    return app.config['DEFER_ON_TLS_ERROR']
NameError: name 'app' is not defined



FROM THE POSTFIX LOG (when we are doing a submit of a mail with the same client we have used since we started the development / use of the service)



Sep 23 23:45:27 mail postfix/smtpd[2583]: connect from 10-196-225-85.secops-mail-service-mailu-front.secops-system.svc.cluster.local[10.196.225.85]
Sep 23 23:45:27 mail postfix/smtpd[2583]: discarding EHLO keywords: PIPELINING
Sep 23 23:45:27 mail postfix/smtpd[2583]: discarding EHLO keywords: PIPELINING
Sep 23 23:45:28 mail postfix/smtpd[2583]: 516185A0C77: client=10-196-226-1.internal-istio-ingressgateway.istio-system.svc.cluster.local[10.196.226.1], sasl_method=XCLIENT, sasl_username=demoadm@mail.
Sep 23 23:45:28 mail postfix/cleanup[2719]: 516185A0C77: replace: header Received: from localhost (10-196-226-1.internal-istio-ingressgateway.istio-system.svc.cluster.local [10.196.226.1])??by mail. (Postfix) with ESMTPA id 516185A0C77??for <lmc from 10-196-226-1.internal-istio-ingressgateway.istio-system.svc.cluster.local[10.196.226.1]; from=<demoadm@systems> to=<lmcdasm@gmail.com> proto=ESMTP helo=<localhost>: Received: from authenticated-user (mail.[10.196.225.152])??by mail. (Postfix) with ESMTPA id 516185A0C77??for <lmcdasm@gmail.com>; Thu, 23 Sep 2021 23:45:28 +0000 (UTC)
Sep 23 23:45:28 mail postfix/cleanup[2719]: 516185A0C77: message-id=<>
Sep 23 23:45:28 mail postfix/qmgr[377]: 516185A0C77: from=<demoadm@.systems>, size=380, nrcpt=1 (queue active)
Sep 23 23:45:28 mail postfix/smtpd[2583]: disconnect from 10-196-226-1.internal-istio-ingressgateway.istio-system.svc.cluster.local[10.196.226.1] ehlo=2 xclient=0/1 mail=1 rcpt=1 data=1 quit=1 commands=6/7
ERROR:root:Error when processing request
Traceback (most recent call last):
  File "/usr/lib/python3.9/site-packages/podop/postfix.py", line 104, in process_request
    result = await table.get(key)
  File "/usr/lib/python3.9/site-packages/podop/table.py", line 35, in get
    raise Exception(request.status)
Exception: 500
Sep 23 23:45:28 mail postfix/smtp[2720]: warning: socketmap:unix:/tmp/podop.socket:dane socketmap server temporary error: unknown error
Sep 23 23:45:28 mail postfix/smtp[2720]: warning: socketmap:unix:/tmp/podop.socket:dane lookup error for "-cloud.mail.protection.outlook.com"
Sep 23 23:45:28 mail postfix/smtp[2720]: warning: smtp_tls_policy_maps, next-hop destination "-cloud.mail.protection.outlook.com": policy table lookup error
Sep 23 23:45:28 mail postfix/smtp[2720]: warning: TLS policy lookup for -cloud.mail.protection.outlook.com/-cloud.mail.protection.outlook.com: client TLS configuration problem
Sep 23 23:45:28 mail postfix/smtp[2720]: warning: TLS policy lookup for -cloud.mail.protection.outlook.com/-cloud.mail.protection.outlook.com: client TLS configuration problem
Sep 23 23:45:28 mail postfix/smtp[2720]: 516185A0C77: to=<lmcdasm@gmail.com>, relay=none, delay=0.8, delays=0.36/0.1/0.34/0, dsn=4.7.5, status=deferred (client TLS configuration problem)






Discussion:

Resolving patch:
diff --git a/core/admin/mailu/utils.py b/core/admin/mailu/utils.py
--- a/core/admin/mailu/utils.py
+++ b/core/admin/mailu/utils.py
@@ -58,7 +58,7 @@ def has_dane_record(domain, timeout=10):
         # we will receive this non-specific exception. The safe behaviour is to
         # accept to defer the email.
         flask.current_app.logger.warn(f'Unable to lookup the TLSA record for {domain}. Is the DNSSEC zone okay on https://dnsviz.net/d/{domain}/dnssec/?')
-        return app.config['DEFER_ON_TLS_ERROR']
+        return flask.current_app.config['DEFER_ON_TLS_ERROR']
     except dns.exception.Timeout:
         flask.current_app.logger.warn(f'Timeout while resolving the TLSA record for {domain} ({timeout}s).')
     except dns.resolver.NXDOMAIN:


Test patch: