Repository: moby/moby (language: go)

Issue:
Encrypted overlay networks don't work on ports other than 4789
### Description

TLDR: When using encrypted overlay network, containers on **different** Swarm nodes can not communicate with each other.

Setup: 2 Swarm manager nodes, each one running a netcat container (service names netcat1 and netcat2) deployed as a stack, each container is constrained to one node. Both attach to external encrypted overlay network called `testenc`

Network is created with:
```
docker network create --opt encrypted -d overlay --attachable testenc
```

Then, from node 1, exec into the container and try to reach node 2:
```
curl -vvv http://netcat2:11111
t/o
```

When not using encrypted network, communication works fine.

Additional set info:
Oracle Linux 9.2 (identical setup also not working on OL v8)
Docker install from Docker upstream repos for Centos
Firewalld is not installed, stock iptables managed by docker
Data path port is changed to 4777 from 4789 due to conflict with vmware on port as documented in some other issues.

Iptables entries seem to be correctly created on both nodes:
```
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
DROP       udp  --  anywhere             anywhere             policy match dir in pol none udp dpt:4777match bpf 177 0 0 0,64 0 0 12,84 0 0 4294967040,21 1 0 1048832,6 0 0 0,6 0 0 4294967295
ACCEPT     udp  --  anywhere             anywhere             policy match dir in pol ipsec udp dpt:4777match bpf 177 0 0 0,64 0 0 12,84 0 0 4294967040,21 1 0 1048832,6 0 0 0,6 0 0 4294967295
```

Documentation states that ESP protocol should be allowed but since iptables is the only firewall and we have no additional external firewalls I assume nothing needs to be done in that regard: https://docs.docker.com/engine/swarm/swarm-tutorial/#open-protocols-and-ports-between-the-hosts

tcpdump for esp protocol does not capture any esp packets on either node:
```
tcpdump -i any -p esp -vv -X -U -w node2.pcap
tcpdump: data link type LINUX_SLL2
dropped privs to tcpdump
tcpdump: listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
0 packets captured
4 packets received by filter
0 packets dropped by kernel
```

Tried everything else from:
https://github.com/moby/moby/issues/30727
https://github.com/moby/moby/issues/30766

### Reproduce

See description

### Expected behavior

Communication between containers works

### docker version

```bash
Client: Docker Engine - Community
 Version:           23.0.6
 API version:       1.42
 Go version:        go1.19.9
 Git commit:        ef23cbc
 Built:             Fri May  5 21:19:37 2023
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          23.0.6
  API version:      1.42 (minimum version 1.12)
  Go version:       go1.19.9
  Git commit:       9dbdbd4
  Built:            Fri May  5 21:18:11 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.21
  GitCommit:        3dce8eb055cbb6872793272b4f20ed16117344f8
 runc:
  Version:          1.1.7
  GitCommit:        v1.1.7-0-g860f061
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
```


### docker info

```bash
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.10.5
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.18.1
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 1
  Running: 1
  Paused: 0
  Stopped: 0
 Images: 1
 Server Version: 23.0.6
 Storage Driver: overlay2
  Backing Filesystem: xfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: active
  NodeID: 4fu4mcmtxzt22e339bg3uia1p
  Is Manager: true
  ClusterID: sj9gs7tglqa8ggvsie2g5pqd3
  Managers: 2
  Nodes: 2
  Default Address Pool: 10.0.0.0/8  
  SubnetSize: 24
  Data Path Port: 4777
  Orchestration:
   Task History Retention Limit: 5
  Raft:
   Snapshot Interval: 10000
   Number of Old Snapshots to Retain: 0
   Heartbeat Tick: 1
   Election Tick: 10
  Dispatcher:
   Heartbeat Period: 5 seconds
  CA Configuration:
   Expiry Duration: 3 months
   Force Rotate: 0
  Autolock Managers: false
  Root Rotation In Progress: false
  Node Address: 10.1.11.29
  Manager Addresses:
   10.1.11.29:2377
   10.1.11.45:2377
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 3dce8eb055cbb6872793272b4f20ed16117344f8
 runc version: v1.1.7-0-g860f061
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 5.14.0-284.11.1.el9_2.x86_64
 Operating System: Oracle Linux Server 9.2
 OSType: linux
 Architecture: x86_64
 CPUs: 1
 Total Memory: 1.42GiB
 Name: localhost.localdomain
 ID: 75521ab2-9371-466c-9fc5-8492efe4b2ff
 Docker Root Dir: /srv/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: Running Swarm in a two-manager configuration. This configuration provides
         no fault tolerance, and poses a high risk to lose control over the cluster.
         Refer to https://docs.docker.com/engine/swarm/admin_guide/ to configure the
         Swarm for fault-tolerance.
```


### Additional Info

docker node ls
```
4fu4mcmtxzt22e339bg3uia1p     docker-swarm-2   Ready     Active         Reachable        23.0.6
7qv3rxbtp45axwdxymjgymbu9 *   docker-swarm-1   Ready     Active         Leader           23.0.6
```

`ip -s xfrm state`
```
src 10.1.11.45 dst 10.1.11.29
	proto esp spi 0xf4faae1b(4110069275) reqid 13681891(0x00d0c4e3) mode transport
	replay-window 0 seq 0x00000000 flag  (0x00000000)
	aead rfc4106(gcm(aes)) 0xa6e7b83e73f8f13d9cd01d9e4428cfb5f4faae1b (160 bits) 64
	anti-replay context: seq 0x0, oseq 0x0, bitmap 0x00000000
	sel src 0.0.0.0/0 dst 0.0.0.0/0 uid 0
	lifetime config:
	  limit: soft (INF)(bytes), hard (INF)(bytes)
	  limit: soft (INF)(packets), hard (INF)(packets)
	  expire add: soft 0(sec), hard 0(sec)
	  expire use: soft 0(sec), hard 0(sec)
	lifetime current:
	  0(bytes), 0(packets)
	  add 2023-05-26 15:51:41 use -
	stats:
	  replay-window 0 replay 0 failed 0
src 10.1.11.45 dst 10.1.11.29
	proto esp spi 0xa8e5da89(2833635977) reqid 13681891(0x00d0c4e3) mode transport
	replay-window 0 seq 0x00000000 flag  (0x00000000)
	aead rfc4106(gcm(aes)) 0x8f162afaf3d9afd1ce6c80ea11358263a8e5da89 (160 bits) 64
	anti-replay context: seq 0x0, oseq 0x0, bitmap 0x00000000
	sel src 0.0.0.0/0 dst 0.0.0.0/0 uid 0
	lifetime config:
	  limit: soft (INF)(bytes), hard (INF)(bytes)
	  limit: soft (INF)(packets), hard (INF)(packets)
	  expire add: soft 0(sec), hard 0(sec)
	  expire use: soft 0(sec), hard 0(sec)
	lifetime current:
	  0(bytes), 0(packets)
	  add 2023-05-26 15:51:41 use -
	stats:
	  replay-window 0 replay 0 failed 0
src 10.1.11.29 dst 10.1.11.45
	proto esp spi 0x4203cf42(1107545922) reqid 13681891(0x00d0c4e3) mode transport
	replay-window 0 seq 0x00000000 flag  (0x00000000)
	aead rfc4106(gcm(aes)) 0x16d7387095fb5646ce76fa748f6108274203cf42 (160 bits) 64
	anti-replay context: seq 0x0, oseq 0x0, bitmap 0x00000000
	sel src 0.0.0.0/0 dst 0.0.0.0/0 uid 0
	lifetime config:
	  limit: soft (INF)(bytes), hard (INF)(bytes)
	  limit: soft (INF)(packets), hard (INF)(packets)
	  expire add: soft 0(sec), hard 0(sec)
	  expire use: soft 0(sec), hard 0(sec)
	lifetime current:
	  0(bytes), 0(packets)
	  add 2023-05-26 15:51:41 use -
	stats:
	  replay-window 0 replay 0 failed 0
src 10.1.11.45 dst 10.1.11.29
	proto esp spi 0xee90bca2(4002462882) reqid 13681891(0x00d0c4e3) mode transport
	replay-window 0 seq 0x00000000 flag  (0x00000000)
	aead rfc4106(gcm(aes)) 0x16d7387095fb5646ce76fa748f610827ee90bca2 (160 bits) 64
	anti-replay context: seq 0x0, oseq 0x0, bitmap 0x00000000
	sel src 0.0.0.0/0 dst 0.0.0.0/0 uid 0
	lifetime config:
	  limit: soft (INF)(bytes), hard (INF)(bytes)
	  limit: soft (INF)(packets), hard (INF)(packets)
	  expire add: soft 0(sec), hard 0(sec)
	  expire use: soft 0(sec), hard 0(sec)
	lifetime current:
	  0(bytes), 0(packets)
	  add 2023-05-26 15:51:41 use -
	stats:
	  replay-window 0 replay 0 failed 0
```

Docker daemon startup log:
```
May 26 15:51:35 localhost.localdomain dockerd[827]: time="2023-05-26T15:51:35.136202872+02:00" level=error msg="agent: session failed" backoff=300ms error="session initiation timed out" module=node/agent node.id=4fu4mcmtxzt22e339bg3uia1p
May 26 15:51:35 localhost.localdomain dockerd[827]: time="2023-05-26T15:51:35.136335419+02:00" level=info msg="manager selected by agent for new session: { }" module=node/agent node.id=4fu4mcmtxzt22e339bg3uia1p
May 26 15:51:35 localhost.localdomain dockerd[827]: time="2023-05-26T15:51:35.136392492+02:00" level=info msg="waiting 221.489695ms before registering session" module=node/agent node.id=4fu4mcmtxzt22e339bg3uia1p
May 26 15:51:35 localhost.localdomain dockerd[827]: time="2023-05-26T15:51:35.298135605+02:00" level=info msg="3930ea0ad9682e55 [term: 2] received a MsgVote message with higher term from 2c169d3d9ef6bbc6 [term: 3]" module=raft node.id=4fu4mcmtxzt22e339bg3uia1p
May 26 15:51:35 localhost.localdomain dockerd[827]: time="2023-05-26T15:51:35.299509870+02:00" level=info msg="3930ea0ad9682e55 became follower at term 3" module=raft node.id=4fu4mcmtxzt22e339bg3uia1p
May 26 15:51:35 localhost.localdomain dockerd[827]: time="2023-05-26T15:51:35.299549333+02:00" level=info msg="3930ea0ad9682e55 [logterm: 2, index: 53, vote: 0] cast MsgVote for 2c169d3d9ef6bbc6 [logterm: 2, index: 53] at term 3" module=raft node.id=4fu4mcmtxzt22e339bg3uia1p
May 26 15:51:35 localhost.localdomain dockerd[827]: time="2023-05-26T15:51:35.309040267+02:00" level=info msg="raft.node: 3930ea0ad9682e55 elected leader 2c169d3d9ef6bbc6 at term 3" module=raft node.id=4fu4mcmtxzt22e339bg3uia1p
May 26 15:51:35 localhost.localdomain dockerd[827]: time="2023-05-26T15:51:35.457707204+02:00" level=info msg="initialized VXLAN UDP port to 4777 "
May 26 15:51:35 localhost.localdomain dockerd[827]: time="2023-05-26T15:51:35.459727407+02:00" level=info msg="Initializing Libnetwork Agent Listen-Addr=0.0.0.0 Local-addr=10.1.11.29 Adv-addr=10.1.11.29 Data-addr= Remote-addr-list=[10.1.11.45] MTU=1500"
May 26 15:51:35 localhost.localdomain dockerd[827]: time="2023-05-26T15:51:35.460018025+02:00" level=info msg="New memberlist node - Node:localhost.localdomain will use memberlist nodeID:2fc62ef3f0a4 with config:&{NodeID:2fc62ef3f0a4 Hostname:localhost.localdomain BindAddr:0.0.0.0 AdvertiseAddr:10.1.11.29 BindPort:0 Keys:[[121 242 176 220 42 161 53 253 51 209 46 210 107 67 224 193] [120 214 164 98 131 168 112 111 145 204 64 64 82 64 9 142] [212 231 49 159 255 31 115 254 109 42 125 178 73 110 84 164]] PacketBufferSize:1400 reapEntryInterval:1800000000000 reapNetworkInterval:1825000000000 rejoinClusterDuration:10000000000 rejoinClusterInterval:60000000000 StatsPrintPeriod:5m0s HealthPrintPeriod:1m0s}"
May 26 15:51:35 localhost.localdomain dockerd[827]: time="2023-05-26T15:51:35.466355059+02:00" level=info msg="Node 2fc62ef3f0a4/10.1.11.29, joined gossip cluster"
May 26 15:51:35 localhost.localdomain dockerd[827]: time="2023-05-26T15:51:35.468183005+02:00" level=info msg="Daemon has completed initialization"
May 26 15:51:35 localhost.localdomain dockerd[827]: time="2023-05-26T15:51:35.493597022+02:00" level=info msg="Node 2fc62ef3f0a4/10.1.11.29, added to nodes list"
May 26 15:51:35 localhost.localdomain dockerd[827]: time="2023-05-26T15:51:35.495485130+02:00" level=info msg="The new bootstrap node list is:[10.1.11.45]"
May 26 15:51:35 localhost.localdomain dockerd[827]: time="2023-05-26T15:51:35.521820707+02:00" level=error msg="Error in joining gossip cluster : could not join node to memberlist: 1 error occurred:\n\t* Failed to join 10.1.11.45:7946: dial tcp 10.1.11.45:7946: connect: connection refused\n\n(join will be retried in background)"
May 26 15:51:35 localhost.localdomain dockerd[827]: time="2023-05-26T15:51:35.563550746+02:00" level=error msg="fatal task error" error="No such container: con2_netcat2.1.6v04wtrdok8eh1t1cqniwpnjg" module=node/agent/taskmanager node.id=4fu4mcmtxzt22e339bg3uia1p service.id=7iunm5n15ho4jqngdwc56xu2v task.id=6v04wtrdok8eh1t1cqniwpnjg
May 26 15:51:35 localhost.localdomain systemd[1]: Started Docker Application Container Engine.
May 26 15:51:35 localhost.localdomain dockerd[827]: time="2023-05-26T15:51:35.642289520+02:00" level=info msg="API listen on /run/docker.sock"
May 26 15:51:35 localhost.localdomain dockerd[827]: time="2023-05-26T15:51:35.683474234+02:00" level=info msg="initialized VXLAN UDP port to 4777 "
May 26 15:51:35 localhost.localdomain dockerd[827]: time="2023-05-26T15:51:35.773371869+02:00" level=info msg="initialized VXLAN UDP port to 4777 "
May 26 15:51:37 localhost.localdomain dockerd[827]: time="2023-05-26T15:51:37.562102981+02:00" level=info msg="Node 6ab9d7b4e730/10.1.11.45, joined gossip cluster"
May 26 15:51:37 localhost.localdomain dockerd[827]: time="2023-05-26T15:51:37.563715970+02:00" level=info msg="Node 6ab9d7b4e730/10.1.11.45, added to nodes list"
May 26 15:56:35 localhost.localdomain dockerd[827]: time="2023-05-26T15:56:35.494213474+02:00" level=info msg="NetworkDB stats localhost.localdomain(2fc62ef3f0a4) - netID:jije10wmwrlzv7u67ca6id8r3 leaving:false netPeers:2 entries:8 Queue qLen:0 netMsg/s:0"
```

network inspect on node 2:
```
docker network inspect testenc
[
    {
        "Name": "testenc",
        "Id": "wy8fpx6q9n9krm5n2twl595fg",
        "Created": "2023-05-26T15:51:35.886740366+02:00",
        "Scope": "swarm",
        "Driver": "overlay",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "10.0.1.0/24",
                    "Gateway": "10.0.1.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": true,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "d704f888653a2f6e7165f8c5b798362121c587527892d9b38a0797e317227d9f": {
                "Name": "con2_netcat2.1.mv2bkm5243kn9urilf4cfgc0c",
                "EndpointID": "22cc37ca15e1299463a6e86897939cc18bfcb6b96de3ae5f65abe929c8a07c67",
                "MacAddress": "02:42:0a:00:01:06",
                "IPv4Address": "10.0.1.6/24",
                "IPv6Address": ""
            },
            "lb-testenc": {
                "Name": "testenc-endpoint",
                "EndpointID": "70c5e980bba2aec400ea31a5a76af1fa1c0e0acfae48ec0c621a0c74ed4def4f",
                "MacAddress": "02:42:0a:00:01:07",
                "IPv4Address": "10.0.1.7/24",
                "IPv6Address": ""
            }
        },
        "Options": {
            "com.docker.network.driver.overlay.vxlanid_list": "4097",
            "encrypted": ""
        },
        "Labels": {},
        "Peers": [
            {
                "Name": "2fc62ef3f0a4",
                "IP": "10.1.11.29"
            },
            {
                "Name": "6ab9d7b4e730",
                "IP": "10.1.11.45"
            }
        ]
    }
]
```
notice a weird encrypted empty string. Also, only local container is listed in containers. Not sure if this is expected, just pointing out what jumps out to me.

Network inspect on node 1:
```
[
    {
        "Name": "testenc",
        "Id": "wy8fpx6q9n9krm5n2twl595fg",
        "Created": "2023-05-26T15:51:37.948062187+02:00",
        "Scope": "swarm",
        "Driver": "overlay",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "10.0.1.0/24",
                    "Gateway": "10.0.1.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": true,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "abd301c3ce852a1b1c2d0f5dd88dc89960d0caa435142ce28f1ba062b6a765ba": {
                "Name": "con1_netcat1.1.u68zlykshrvh0exj0myaccyzw",
                "EndpointID": "f293960e5211b0396d9d8b786ea5dcaad72efb8909ab0632f1bfc009ece52104",
                "MacAddress": "02:42:0a:00:01:08",
                "IPv4Address": "10.0.1.8/24",
                "IPv6Address": ""
            },
            "lb-testenc": {
                "Name": "testenc-endpoint",
                "EndpointID": "320cbb2efdc41d88bbbc04c2dc619c43c679b134a83bae8502166af82e85b3c4",
                "MacAddress": "02:42:0a:00:01:09",
                "IPv4Address": "10.0.1.9/24",
                "IPv6Address": ""
            }
        },
        "Options": {
            "com.docker.network.driver.overlay.vxlanid_list": "4097",
            "encrypted": ""
        },
        "Labels": {},
        "Peers": [
            {
                "Name": "2fc62ef3f0a4",
                "IP": "10.1.11.29"
            },
            {
                "Name": "6ab9d7b4e730",
                "IP": "10.1.11.45"
            }
        ]
    }
]
```


Discussion / hints:
> notice a weird encrypted empty string

Yes, that's normal. The presence of the `encrypted` network option is enough to enable encryption; the option value doesn't matter.

Your `docker info` output lists the data path port as 4777, as do your daemon logs
```
May 26 15:51:35 localhost.localdomain dockerd[827]: time="2023-05-26T15:51:35.457707204+02:00" level=info msg="initialized VXLAN UDP port to 4777 "
```
yet your iptables rules are for port 4888. How did that mismatch happen?

> Documentation states that ESP protocol should be allowed but since iptables is the only firewall and we have no additional external firewalls I assume nothing needs to be done in that regard

Given that you aren't seeing any ESP traffic in tcpdump, that may not be a safe assumption. Your iptables input chain might be configured with `policy DROP` or have an unconditional DROP rule at the bottom.
Port is indeed 4777, I mispasted. Corrected the info.

My understanding is that tcpdump will capture before iptables. Anyway, here is `iptables -L` on node 1

```
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
DROP       udp  --  anywhere             anywhere             policy match dir in pol none udp dpt:4777match bpf 177 0 0 0,64 0 0 12,84 0 0 4294967040,21 1 0 1048832,6 0 0 0,6 0 0 4294967295
ACCEPT     udp  --  anywhere             anywhere             policy match dir in pol ipsec udp dpt:4777match bpf 177 0 0 0,64 0 0 12,84 0 0 4294967040,21 1 0 1048832,6 0 0 0,6 0 0 4294967295

Chain FORWARD (policy DROP)
target     prot opt source               destination         
DOCKER-USER  all  --  anywhere             anywhere            
DOCKER-INGRESS  all  --  anywhere             anywhere            
DOCKER-ISOLATION-STAGE-1  all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
DROP       all  --  anywhere             anywhere            

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

Chain DOCKER (2 references)
target     prot opt source               destination         

Chain DOCKER-INGRESS (1 references)
target     prot opt source               destination         
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:19111
ACCEPT     tcp  --  anywhere             anywhere             state RELATED,ESTABLISHED tcp spt:19111
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:19112
ACCEPT     tcp  --  anywhere             anywhere             state RELATED,ESTABLISHED tcp spt:19112
RETURN     all  --  anywhere             anywhere            

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
target     prot opt source               destination         
DOCKER-ISOLATION-STAGE-2  all  --  anywhere             anywhere            
DOCKER-ISOLATION-STAGE-2  all  --  anywhere             anywhere            
RETURN     all  --  anywhere             anywhere            

Chain DOCKER-ISOLATION-STAGE-2 (2 references)
target     prot opt source               destination         
DROP       all  --  anywhere             anywhere            
DROP       all  --  anywhere             anywhere            
RETURN     all  --  anywhere             anywhere            

Chain DOCKER-USER (1 references)
target     prot opt source               destination         
RETURN     all  --  anywhere             anywhere 
```
> My understanding is that tcpdump will capture before iptables.

Whoops, you're right. Could you please post the output of `iptables-save`? I need to check the contents of the other tables.
```
# Generated by iptables-save v1.8.8 (nf_tables) on Fri May 26 17:25:15 2023
*mangle
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
-A OUTPUT -p udp -m udp --dport 4777 -m bpf --bytecode "6,177 0 0 0,64 0 0 12,84 0 0 4294967040,21 1 0 1048832,6 0 0 0,6 0 0 4294967295" -j MARK --set-xmark 0xd0c4e3/0xffffffff
COMMIT
# Completed on Fri May 26 17:25:15 2023
# Generated by iptables-save v1.8.8 (nf_tables) on Fri May 26 17:25:15 2023
*filter
:INPUT ACCEPT [0:0]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [0:0]
:DOCKER - [0:0]
:DOCKER-INGRESS - [0:0]
:DOCKER-ISOLATION-STAGE-1 - [0:0]
:DOCKER-ISOLATION-STAGE-2 - [0:0]
:DOCKER-USER - [0:0]
-A INPUT -p udp -m policy --dir in --pol none -m udp --dport 4777 -m bpf --bytecode "6,177 0 0 0,64 0 0 12,84 0 0 4294967040,21 1 0 1048832,6 0 0 0,6 0 0 4294967295" -j DROP
-A INPUT -p udp -m policy --dir in --pol ipsec -m udp --dport 4777 -m bpf --bytecode "6,177 0 0 0,64 0 0 12,84 0 0 4294967040,21 1 0 1048832,6 0 0 0,6 0 0 4294967295" -j ACCEPT
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-INGRESS
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A FORWARD -o docker_gwbridge -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker_gwbridge -j DOCKER
-A FORWARD -i docker_gwbridge ! -o docker_gwbridge -j ACCEPT
-A FORWARD -i docker_gwbridge -o docker_gwbridge -j DROP
-A DOCKER-INGRESS -p tcp -m tcp --dport 19111 -j ACCEPT
-A DOCKER-INGRESS -p tcp -m state --state RELATED,ESTABLISHED -m tcp --sport 19111 -j ACCEPT
-A DOCKER-INGRESS -p tcp -m tcp --dport 19112 -j ACCEPT
-A DOCKER-INGRESS -p tcp -m state --state RELATED,ESTABLISHED -m tcp --sport 19112 -j ACCEPT
-A DOCKER-INGRESS -j RETURN
-A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -i docker_gwbridge ! -o docker_gwbridge -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -j RETURN
-A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -o docker_gwbridge -j DROP
-A DOCKER-ISOLATION-STAGE-2 -j RETURN
-A DOCKER-USER -j RETURN
COMMIT
# Completed on Fri May 26 17:25:15 2023
# Generated by iptables-save v1.8.8 (nf_tables) on Fri May 26 17:25:15 2023
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:DOCKER - [0:0]
:DOCKER-INGRESS - [0:0]
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER-INGRESS
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT -m addrtype --dst-type LOCAL -j DOCKER-INGRESS
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -o docker_gwbridge -m addrtype --src-type LOCAL -j MASQUERADE
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A POSTROUTING -s 172.18.0.0/16 ! -o docker_gwbridge -j MASQUERADE
-A DOCKER -i docker0 -j RETURN
-A DOCKER -i docker_gwbridge -j RETURN
-A DOCKER-INGRESS -p tcp -m tcp --dport 19111 -j DNAT --to-destination 172.18.0.2:19111
-A DOCKER-INGRESS -p tcp -m tcp --dport 19112 -j DNAT --to-destination 172.18.0.2:19112
-A DOCKER-INGRESS -j RETURN
COMMIT
```
Hmm, I don't see anything out of place in your iptables rules. Is the output of `ip -s xfrm state` from node1, the node running the container from which you tried to make an outgoing request? The packet counters being all zeroes suggests that no packets outgoing or incoming are having IPSec transformations applied. Where are the packets going instead?

I see that your Swarm nodes have IP addresses in the range 10.1.11.x, and the overlay network is assigned the 10.0.0.0/24 subnet. What is the subnet mask of your host network? If the two networks overlap, perhaps the packets from the container are being routed out onto the host network instead of through the overlay VXLAN. And just for completeness, mind checking if tcpdump captures any packets on 4777/udp?
`ip -s xfrm state` on both nodes shows 0 packets.

Host subnet is `10.1.11.0/24` so should be fine. Will try a different subnet just in case.

There is some UDP 4777 traffic (node1 -> node2)

<img width="933" alt="Screenshot 2023-05-26 at 18 19 04" src="https://github.com/moby/moby/assets/1158861/e8e7db90-4c60-435e-bade-208a40b196ae">

Using `--subnet=172.31.0.0/24` did not help. Tried same subnet without encryption, no problems.

Surprisingly, name resolution IPs seem to be off but maybe I am reading it wrong.

```
curl -vvv http://netcat2:11111
*   Trying 172.31.0.2:11111
```

But docker inspect of netcat2 shows
```
...
"Networks": {
"testenc": {
                    "IPAddress": "172.31.0.3",
```

Similarly, self test from inside netcat1 `curl -vvv http://netcat1:11111` (which connects and returns fine obviously) resolves to `172.31.0.5` while inspect of netcat1 shows `172.31.0.6`. Must be another layer on top I am not understanding.
Yep, that's definitely a VXLAN packet tagged with VNI 4097, encapsulating a TCP segment. Do you see the UDP 4777 packets actually being routed to node2? The rule
```
iptables -t filter -A INPUT -p udp -m policy --dir in --pol none -m udp --dport 4777 -m bpf --bytecode [...] -j DROP
```
would filter out those packets from being decapsulated and forwarded to the container if they did get received, though they would still show up in a packet capture.

On the sending node, could you check the counters on the `-t mangle -A OUTPUT [...] --set-xmark` rule? (`iptables -t mangle -vL OUTPUT`) That's the rule which tags outgoing packets for IPSec processing.

The output of `ip -s xfrm policy` might also help.
Ok, major "breakthrough", using port 4789 on swarm init works but changing the port breaks it. However, this I can not do in targeted production environment due to the mentioned vmware issue. This post from the poor soulmate got me the idea: https://forums.docker.com/t/overlay-network-with-encryption-and-changed-data-path-port/122348



So the TLDR is: using a custom data path port breaks encrypted overlay networks


I am ending the shift for today but will run what you suggested over the weekend.
> Ok, major "breakthrough", using port 4789 on swarm init works but changing the port breaks it.

🤦 yep, [that's a bug.](https://github.com/moby/moby/blob/13fb24458c8af5be85472c38ffcd1349df303d9a/libnetwork/drivers/overlay/encryption.go#L372)

Fix patch:
diff --git a/libnetwork/drivers/overlay/encryption.go b/libnetwork/drivers/overlay/encryption.go
index 79f7169b6d949..a9104d1820469 100644
--- a/libnetwork/drivers/overlay/encryption.go
+++ b/libnetwork/drivers/overlay/encryption.go
@@ -375,8 +375,8 @@ func programSP(fSA *netlink.XfrmState, rSA *netlink.XfrmState, add bool) error {
 		Src:     &net.IPNet{IP: s, Mask: fullMask},
 		Dst:     &net.IPNet{IP: d, Mask: fullMask},
 		Dir:     netlink.XFRM_DIR_OUT,
-		Proto:   17,
-		DstPort: 4789,
+		Proto:   syscall.IPPROTO_UDP,
+		DstPort: int(overlayutils.VXLANUDPPort()),
 		Mark:    &spMark,
 		Tmpls: []netlink.XfrmPolicyTmpl{
 			{
@@ -589,8 +589,8 @@ func updateNodeKey(lIP, aIP, rIP net.IP, idxs []*spi, curKeys []*key, newIdx, pr
 			Src:     &net.IPNet{IP: s, Mask: fullMask},
 			Dst:     &net.IPNet{IP: d, Mask: fullMask},
 			Dir:     netlink.XFRM_DIR_OUT,
-			Proto:   17,
-			DstPort: 4789,
+			Proto:   syscall.IPPROTO_UDP,
+			DstPort: int(overlayutils.VXLANUDPPort()),
 			Mark:    &spMark,
 			Tmpls: []netlink.XfrmPolicyTmpl{
 				{


Test patch: