Repository: moby/moby (language: go)

Issue:
Something is wrong with networking, perhaps DNS in 24.0, unexplained delays
### Description

I'm sorry that I do not have enough expertise to get to the bottom of this, and that I'm using third party software to demonstrate the issue. I'm sure this can be narrowed down to something more specific, but I'm unable to do that.

My postfix docker image after upgrading to docker 24.0 introduces a 5 second delay between a client is connected to it and the smtp banner is returned. The same image is working fine on 23.0. The delay is causing the Load balance to fail the health check and put the mail server out of action. Some SMTP clients do not like the delay either.

I know that postfix before displaying the banner attempt to run a reverse DNS query on the incoming IP, possible also some other DNS queries. This could cause the observed delay.

### Reproduce

Operating system: **Ubuntu 22.04 LTS**.

Install docker:

```
sudo apt-get update
sudo apt-get install ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
sudo chmod a+r /etc/apt/keyrings/docker.gpg
echo \
  "deb [arch="$(dpkg --print-architecture)" signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
  "$(. /etc/os-release && echo "$VERSION_CODENAME")" stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update
sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
```

**Dockerfile**:

```
FROM alpine:3.17.3
RUN apk add --no-cache postfix postfix-pcre cyrus-sasl cyrus-sasl-login cyrus-sasl-crammd5 tzdata
HEALTHCHECK --start-period=350s CMD echo QUIT|nc localhost 587|grep "220 .* ESMTP Postfix"
CMD ["sh","-c","postconf -e 'readme_directory = no' && postfix set-permissions && postfix start-fg"]
```

Build with:

```
docker build . -t postfix
```
Postfix configuration above is not a working postfix configuration, it is minimised so I could reproduce the problem. This post won't send or receive emails and is not secure.

**postfix\master.cf**:

```
# Postfix master process configuration file.  For details on the format
# of the file, see the master(5) manual page (command: "man 5 master" or
# on-line: http://www.postfix.org/master.5.html).
#
# Do not forget to execute "postfix reload" after editing this file.
#
# ==========================================================================
# service type  private unpriv  chroot  wakeup  maxproc command + args
#               (yes)   (yes)   (no)    (never) (100)
# ==========================================================================
#smtp      inet  n       -       n       -       -       smtpd
submission inet  n       -       n       -       -       smtpd
#628       inet  n       -       n       -       -       qmqpd
pickup    unix  n       -       n       60      1       pickup
cleanup   unix  n       -       n       -       0       cleanup
qmgr      unix  n       -       n       300     1       qmgr
#qmgr     unix  n       -       n       300     1       oqmgr
tlsmgr    unix  -       -       n       1000?   1       tlsmgr
rewrite   unix  -       -       n       -       -       trivial-rewrite
bounce    unix  -       -       n       -       0       bounce
defer     unix  -       -       n       -       0       bounce
trace     unix  -       -       n       -       0       bounce
verify    unix  -       -       n       -       1       verify
flush     unix  n       -       n       1000?   0       flush
proxymap  unix  -       -       n       -       -       proxymap
proxywrite unix -       -       n       -       1       proxymap
smtp      unix  -       -       n       -       -       smtp
relay     unix  -       -       n       -       -       smtp
        -o syslog_name=postfix/$service_name
#       -o smtp_helo_timeout=5 -o smtp_connect_timeout=5
showq     unix  n       -       n       -       -       showq
error     unix  -       -       n       -       -       error
retry     unix  -       -       n       -       -       error
discard   unix  -       -       n       -       -       discard
local     unix  -       n       n       -       -       local
virtual   unix  -       n       n       -       -       virtual
lmtp      unix  -       -       n       -       -       lmtp
anvil     unix  -       -       n       -       1       anvil
scache    unix  -       -       n       -       1       scache
postlog   unix-dgram n  -       n       -       1       postlogd
```

**postfix/main.cf**:

```
maillog_file  =  /dev/stdout
compatibility_level = 3.6
readme_directory = no
smtpd_banner = $myhostname ESMTP $mail_name $mail_version
smtpd_recipient_restrictions = reject_unauth_destination
```

**init.sh**:

```
#!/bin/bash

set -e
SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )

rm -rf /mnt/docker/postfix
mkdir -p /mnt/docker/postfix

docker run --rm -d --name postfix_config -v "$SCRIPT_DIR/postfix":/config postfix
docker exec -it postfix_config postalias /etc/postfix/aliases
docker cp postfix_config:/etc/postfix /mnt/docker/postfix
mv /mnt/docker/postfix/postfix /mnt/docker/postfix/config
docker stop postfix_config

cp -r "$SCRIPT_DIR/postfix"/* /mnt/docker/postfix/config

chown 0:0 -R /mnt/docker/postfix/config
chmod -R 600 /mnt/docker/postfix/config
```
To configure run:

```
sudo ./init.sh
```

**docker-compose.yaml**

```
services:
  postfix:
    image: postfix
    container_name: postfix
    init: true
    volumes:
      - /mnt/docker/postfix/config:/etc/postfix
    environment:
      - TZ=Pacific/Auckland
    ports:
      - 587:587
    logging:
      driver: local
```

Run:

```
docker compose up -d --force-recreate
```

Now from a *different machine* try to telnet there:

```
telnet aaa.bbb.ccc.ddd 587
```

Observe that the banner displayed with the 5 second delay.

```
220 7595ee2ac990.localdomain ESMTP Postfix 3.7.4
```

### Expected behavior

After doing the above downgradethe docker:

```
sudo apt install docker-ce=5:23.0.6-1~ubuntu.22.04~jammy
```

And re-run docker compose:

```
sudo docker compose up -d --force-recreate
```

And telnet from different machine:

```
telnet aaa.bbb.ccc.ddd 587
```

observe that the banner is displayed immediately

### docker version

```bash
Client: Docker Engine - Community
 Version:           24.0.0
 API version:       1.43
 Go version:        go1.20.4
 Git commit:        98fdcd7
 Built:             Mon May 15 18:49:22 2023
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          24.0.0
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.4
  Git commit:       1331b8c
  Built:            Mon May 15 18:49:22 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.21
  GitCommit:        3dce8eb055cbb6872793272b4f20ed16117344f8
 runc:
  Version:          1.1.7
  GitCommit:        v1.1.7-0-g860f061
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
```


### docker info

```bash
Client: Docker Engine - Community
 Version:    24.0.0
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.10.4
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.17.3
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 1
  Running: 0
  Paused: 0
  Stopped: 1
 Images: 1
 Server Version: 24.0.0
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 3dce8eb055cbb6872793272b4f20ed16117344f8
 runc version: v1.1.7-0-g860f061
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 5.15.0-72-generic
 Operating System: Ubuntu 22.04.2 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 2
 Total Memory: 3.832GiB
 Name: d-in-andrew-10
 ID: 42805cb8-2959-4b77-a2bb-8b495593716e
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
```


### Additional Info

_No response_


Discussion / hints:
Probably related, DNS is broken in `alpine`-based images ran in `24.0.0` when using a non-default network.

Works fine in `23.0.6` or when using the default `bridge` network.

Looking at the changelog, this seems to have been introduced by [#44664](https://github.com/moby/moby/pull/44664) **libnetwork: improve embedded DNS resolver**.

@AndrewSav You could try running it using the default `bridge` network:

```yaml
services:
  postfix:
    image: postfix
    network_mode: bridge
    # [...]
```

or use a different base image than `alpine`.

If the above assumption is correct, this should work fine.
@lbogdan, thank you very much, yes `network_mode: bridge` works around it.

Fix patch:
diff --git a/libnetwork/resolver.go b/libnetwork/resolver.go
index e8e694403953c..1304e95df2a30 100644
--- a/libnetwork/resolver.go
+++ b/libnetwork/resolver.go
@@ -449,7 +449,6 @@ func (r *Resolver) dialExtDNS(proto string, server extDNSEntry) (net.Conn, error
 }
 
 func (r *Resolver) forwardExtDNS(proto string, query *dns.Msg) *dns.Msg {
-	queryName, queryType := query.Question[0].Name, query.Question[0].Qtype
 	for _, extDNS := range r.extDNSList {
 		if extDNS.IPStr == "" {
 			break
@@ -477,20 +476,7 @@ func (r *Resolver) forwardExtDNS(proto string, query *dns.Msg) *dns.Msg {
 		case dns.RcodeServerFailure, dns.RcodeRefused:
 			// Server returned FAILURE: continue with the next external DNS server
 			// Server returned REFUSED: this can be a transitional status, so continue with the next external DNS server
-			logrus.Debugf("[resolver] external DNS %s:%s responded with %s for %q", proto, extDNS.IPStr, statusString(resp.Rcode), queryName)
-			continue
-		case dns.RcodeNameError:
-			// Server returned NXDOMAIN. Stop resolution if it's an authoritative answer (see RFC 8020: https://tools.ietf.org/html/rfc8020#section-2)
-			logrus.Debugf("[resolver] external DNS %s:%s responded with %s for %q", proto, extDNS.IPStr, statusString(resp.Rcode), queryName)
-			if resp.Authoritative {
-				break
-			}
-			continue
-		case dns.RcodeSuccess:
-			// All is well
-		default:
-			// Server gave some error. Log the error, and continue with the next external DNS server
-			logrus.Debugf("[resolver] external DNS %s:%s responded with %s (code %d) for %q", proto, extDNS.IPStr, statusString(resp.Rcode), resp.Rcode, queryName)
+			logrus.Debugf("[resolver] external DNS %s:%s returned failure:\n%s", proto, extDNS.IPStr, resp)
 			continue
 		}
 		answers := 0
@@ -509,8 +495,8 @@ func (r *Resolver) forwardExtDNS(proto string, query *dns.Msg) *dns.Msg {
 				r.backend.HandleQueryResp(h.Name, ip)
 			}
 		}
-		if resp.Answer == nil || answers == 0 {
-			logrus.Debugf("[resolver] external DNS %s:%s did not return any %s records for %q", proto, extDNS.IPStr, dns.TypeToString[queryType], queryName)
+		if len(resp.Answer) == 0 {
+			logrus.Debugf("[resolver] external DNS %s:%s returned response with no answers:\n%s", proto, extDNS.IPStr, resp)
 		}
 		resp.Compress = true
 		return resp
@@ -558,10 +544,3 @@ func (r *Resolver) exchange(proto string, extDNS extDNSEntry, query *dns.Msg) *d
 	}
 	return resp
 }
-
-func statusString(responseCode int) string {
-	if s, ok := dns.RcodeToString[responseCode]; ok {
-		return s
-	}
-	return "UNKNOWN"
-}


Test patch:
diff --git a/libnetwork/resolver_test.go b/libnetwork/resolver_test.go
index e782de6e2a3c9..4637a4298ebbd 100644
--- a/libnetwork/resolver_test.go
+++ b/libnetwork/resolver_test.go
@@ -13,6 +13,7 @@ import (
 	"github.com/miekg/dns"
 	"github.com/sirupsen/logrus"
 	"gotest.tools/v3/assert"
+	is "gotest.tools/v3/assert/cmp"
 	"gotest.tools/v3/skip"
 )
 
@@ -457,3 +458,60 @@ type badSRVDNSBackend struct{ noopDNSBackend }
 func (badSRVDNSBackend) ResolveService(name string) ([]*net.SRV, []net.IP) {
 	return []*net.SRV{nil, nil, nil}, nil // Mismatched slice lengths
 }
+
+func TestProxyNXDOMAIN(t *testing.T) {
+	mockSOA, err := dns.NewRR(".	86367	IN	SOA	a.root-servers.net. nstld.verisign-grs.com. 2023051800 1800 900 604800 86400\n")
+	assert.NilError(t, err)
+	assert.Assert(t, mockSOA != nil)
+
+	serveStarted := make(chan struct{})
+	srv := &dns.Server{
+		Net:  "udp",
+		Addr: "127.0.0.1:0",
+		Handler: dns.HandlerFunc(func(w dns.ResponseWriter, r *dns.Msg) {
+			msg := new(dns.Msg).SetRcode(r, dns.RcodeNameError)
+			msg.Ns = append(msg.Ns, dns.Copy(mockSOA))
+			w.WriteMsg(msg)
+		}),
+		NotifyStartedFunc: func() { close(serveStarted) },
+	}
+	serveDone := make(chan error, 1)
+	go func() {
+		defer close(serveDone)
+		serveDone <- srv.ListenAndServe()
+	}()
+
+	select {
+	case err := <-serveDone:
+		t.Fatal(err)
+	case <-serveStarted:
+	}
+
+	defer func() {
+		if err := srv.Shutdown(); err != nil {
+			t.Error(err)
+		}
+		<-serveDone
+	}()
+
+	srvAddr := srv.PacketConn.LocalAddr().(*net.UDPAddr)
+	rsv := NewResolver("", true, noopDNSBackend{})
+	rsv.SetExtServers([]extDNSEntry{
+		{IPStr: srvAddr.IP.String(), port: uint16(srvAddr.Port), HostLoopback: true},
+	})
+
+	// The resolver logs lots of valuable info at level debug. Redirect it
+	// to t.Log() so the log spew is emitted only if the test fails.
+	defer redirectLogrusTo(t)()
+
+	w := &tstwriter{localAddr: srv.PacketConn.LocalAddr()}
+	q := new(dns.Msg).SetQuestion("example.net.", dns.TypeA)
+	rsv.serveDNS(w, q)
+	resp := w.GetResponse()
+	checkNonNullResponse(t, resp)
+	t.Log("Response:\n" + resp.String())
+	checkDNSResponseCode(t, resp, dns.RcodeNameError)
+	assert.Assert(t, is.Len(resp.Answer, 0))
+	assert.Assert(t, is.Len(resp.Ns, 1))
+	assert.Equal(t, resp.Ns[0].String(), mockSOA.String())
+}