Repository: moby/moby (language: go)

Issue:
Docker swarm does not send VXLAN traffic when configured to use IPv6
<!--
If you are reporting a new issue, make sure that we do not have any duplicates
already open. You can ensure this by searching the issue list for this
repository. If there is a duplicate, please close your issue and add a comment
to the existing issue instead.

If you suspect your issue is a bug, please edit your issue description to
include the BUG REPORT INFORMATION shown below. If you fail to provide this
information within 7 days, we cannot debug your issue and will close it. We
will, however, reopen it if you later provide the information.

For more information about reporting issues, see
https://github.com/moby/moby/blob/master/CONTRIBUTING.md#reporting-other-issues

---------------------------------------------------
GENERAL SUPPORT INFORMATION
---------------------------------------------------

The GitHub issue tracker is for bug reports and feature requests.
General support for **docker** can be found at the following locations:

- Docker Support Forums - https://forums.docker.com
- Slack - community.docker.com #general channel
- Post a question on StackOverflow, using the Docker tag

General support for **moby** can be found at the following locations:

- Moby Project Forums - https://forums.mobyproject.org
- Slack - community.docker.com #moby-project channel
- Post a question on StackOverflow, using the Moby tag

---------------------------------------------------
BUG REPORT INFORMATION
---------------------------------------------------
Use the commands below to provide key information from your environment:
You do NOT have to include this information if this is a FEATURE REQUEST
-->

**Description**

<!--
Briefly describe the problem you are having in a few paragraphs.
-->

Using [this guide](https://docs.docker.com/network/network-tutorial-overlay/) to setup overlay networking between two standalone containers on different hosts, everything works up until the point that I use ping to test connectivity between the containers, which does not work -- DNS resolution works fine, but the containers cannot communicate.

The issue occurs when the swarm is initiated using an IPv6 address: `docker swarm init --advertise-addr <IPv6::addr>`.  `tcpdump` on both hosts shows that packets are flowing correctly over the IPv6 interface on ports 2377 and 7946, but there is no traffic at all on the VXLAN port 4789 (IPv6 or IPv4).

There are no firewalls on either host and testing using `netcat` with Docker shutdown shows that there are no issues communicating between hosts on port 4789, both IPv6 and IPv4.

Following the guide again, but forcing Docker to use the IPv4 interface (`docker swarm init --advertise-addr <IPv4.addr>`) makes everything work properly, so this appears to be only an IPv6 issue.

**Steps to reproduce the issue:**
1. Follow [this guide](https://docs.docker.com/network/network-tutorial-overlay/)
2. Specify an IPv6 address for the `docker swarm init --advertise-addr` command
3. Try to ping between containers

**Describe the results you received:**

There is no network connectivity between the containers.

**Describe the results you expected:**

There should be network connectivity between the containers.

**Additional information you deem important (e.g. issue happens only occasionally):**

**Output of `docker version`:**

```
Client: Docker Engine - Community
 Version:           19.03.13
 API version:       1.40
 Go version:        go1.13.15
 Git commit:        4484c46d9d
 Built:             Wed Sep 16 17:02:52 2020
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.13
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       4484c46d9d
  Built:            Wed Sep 16 17:01:20 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.3.7
  GitCommit:        8fba4e9a7d01810a393d5d25a3621dc101981175
 runc:
  Version:          1.0.0-rc10
  GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683
```

**Output of `docker info`:**

```
Client:
 Debug Mode: false

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 1
 Server Version: 19.03.13
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 8fba4e9a7d01810a393d5d25a3621dc101981175
 runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
 init version: fec3683
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 5.4.0-54-generic
 Operating System: Ubuntu 20.04.1 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 2
 Total Memory: 1.941GiB
 Name: beta
 ID: BSTM:JVK3:KORD:JEZ5:CTS5:NSLV:OWX5:MM7W:2L7J:3CHP:TO6Y:S4B6
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No swap limit support
```

**Additional environment details (AWS, VirtualBox, physical, etc.):**

The hosts are Xen VPS instances.  OS is Ubuntu 20.04, but I have also tried Ubuntu 18.04 with the same Docker version.


Discussion / hints:
Can confirm that, spend 10 hours trying to fix it... :/

It would be perfect if it would be working.
So private ipv6 addresses without a fixed ipv4 could join a docker swarm. :)

Fix patch:
diff --git a/libnetwork/drivers/overlay/encryption.go b/libnetwork/drivers/overlay/encryption.go
index e2eefbeeb2222..a2cf51bac991e 100644
--- a/libnetwork/drivers/overlay/encryption.go
+++ b/libnetwork/drivers/overlay/encryption.go
@@ -124,8 +124,8 @@ func (d *driver) checkEncryption(nid string, rIP net.IP, isLocal, add bool) erro
 		return types.ForbiddenErrorf("encryption key is not present")
 	}
 
-	lIP := net.ParseIP(d.bindAddress)
-	aIP := net.ParseIP(d.advertiseAddress)
+	lIP := d.bindAddress
+	aIP := d.advertiseAddress
 	nodes := map[string]net.IP{}
 
 	switch {
@@ -225,7 +225,19 @@ func removeEncryption(localIP, remoteIP net.IP, em *encrMap) error {
 	return nil
 }
 
-func programMangle(vni uint32, add bool) error {
+func (d *driver) transportIPTable() (*iptables.IPTable, error) {
+	v6, err := d.isIPv6Transport()
+	if err != nil {
+		return nil, err
+	}
+	version := iptables.IPv4
+	if v6 {
+		version = iptables.IPv6
+	}
+	return iptables.GetIptable(version), nil
+}
+
+func (d *driver) programMangle(vni uint32, add bool) error {
 	var (
 		m      = strconv.FormatUint(mark, 10)
 		chain  = "OUTPUT"
@@ -234,8 +246,11 @@ func programMangle(vni uint32, add bool) error {
 		action = "install"
 	)
 
-	// TODO IPv6 support
-	iptable := iptables.GetIptable(iptables.IPv4)
+	iptable, err := d.transportIPTable()
+	if err != nil {
+		// Fail closed if unsure. Better safe than cleartext.
+		return err
+	}
 
 	if !add {
 		a = iptables.Delete
@@ -249,7 +264,7 @@ func programMangle(vni uint32, add bool) error {
 	return nil
 }
 
-func programInput(vni uint32, add bool) error {
+func (d *driver) programInput(vni uint32, add bool) error {
 	var (
 		plainVxlan = matchVXLAN(overlayutils.VXLANUDPPort(), vni)
 		chain      = "INPUT"
@@ -261,8 +276,11 @@ func programInput(vni uint32, add bool) error {
 		return append(args, "-j", jump)
 	}
 
-	// TODO IPv6 support
-	iptable := iptables.GetIptable(iptables.IPv4)
+	iptable, err := d.transportIPTable()
+	if err != nil {
+		// Fail closed if unsure. Better safe than cleartext.
+		return err
+	}
 
 	if !add {
 		msg = "remove"
@@ -495,8 +513,8 @@ func (d *driver) updateKeys(newKey, primary, pruneKey *key) error {
 		newIdx = -1
 		priIdx = -1
 		delIdx = -1
-		lIP    = net.ParseIP(d.bindAddress)
-		aIP    = net.ParseIP(d.advertiseAddress)
+		lIP    = d.bindAddress
+		aIP    = d.advertiseAddress
 	)
 
 	d.Lock()
diff --git a/libnetwork/drivers/overlay/joinleave.go b/libnetwork/drivers/overlay/joinleave.go
index 26c8e8aea73d6..c67470ffec5b6 100644
--- a/libnetwork/drivers/overlay/joinleave.go
+++ b/libnetwork/drivers/overlay/joinleave.go
@@ -107,7 +107,7 @@ func (d *driver) Join(nid, eid string, sboxKey string, jinfo driverapi.JoinInfo,
 		}
 	}
 
-	d.peerAdd(nid, eid, ep.addr.IP, ep.addr.Mask, ep.mac, net.ParseIP(d.advertiseAddress), false, false, true)
+	d.peerAdd(nid, eid, ep.addr.IP, ep.addr.Mask, ep.mac, d.advertiseAddress, false, false, true)
 
 	if err = d.checkEncryption(nid, nil, true, true); err != nil {
 		log.G(context.TODO()).Warn(err)
@@ -116,7 +116,7 @@ func (d *driver) Join(nid, eid string, sboxKey string, jinfo driverapi.JoinInfo,
 	buf, err := proto.Marshal(&PeerRecord{
 		EndpointIP:       ep.addr.String(),
 		EndpointMAC:      ep.mac.String(),
-		TunnelEndpointIP: d.advertiseAddress,
+		TunnelEndpointIP: d.advertiseAddress.String(),
 	})
 	if err != nil {
 		return err
@@ -162,7 +162,7 @@ func (d *driver) EventNotify(etype driverapi.EventType, nid, tableName, key stri
 
 	// Ignore local peers. We already know about them and they
 	// should not be added to vxlan fdb.
-	if peer.TunnelEndpointIP == d.advertiseAddress {
+	if net.ParseIP(peer.TunnelEndpointIP).Equal(d.advertiseAddress) {
 		return
 	}
 
@@ -209,7 +209,7 @@ func (d *driver) Leave(nid, eid string) error {
 		return types.InternalMaskableErrorf("could not find endpoint with id %s", eid)
 	}
 
-	d.peerDelete(nid, eid, ep.addr.IP, ep.addr.Mask, ep.mac, net.ParseIP(d.advertiseAddress), true)
+	d.peerDelete(nid, eid, ep.addr.IP, ep.addr.Mask, ep.mac, d.advertiseAddress, true)
 
 	n.leaveSandbox()
 
diff --git a/libnetwork/drivers/overlay/ov_network.go b/libnetwork/drivers/overlay/ov_network.go
index 2e475152a0710..15ad20d2e62a9 100644
--- a/libnetwork/drivers/overlay/ov_network.go
+++ b/libnetwork/drivers/overlay/ov_network.go
@@ -155,8 +155,8 @@ func (d *driver) CreateNetwork(id string, option map[string]interface{}, nInfo d
 	// Make sure no rule is on the way from any stale secure network
 	if !n.secure {
 		for _, vni := range vnis {
-			programMangle(vni, false)
-			programInput(vni, false)
+			d.programMangle(vni, false)
+			d.programInput(vni, false)
 		}
 	}
 
@@ -215,14 +215,14 @@ func (d *driver) DeleteNetwork(nid string) error {
 
 	if n.secure {
 		for _, s := range n.subnets {
-			if err := programMangle(s.vni, false); err != nil {
+			if err := d.programMangle(s.vni, false); err != nil {
 				log.G(context.TODO()).WithFields(log.Fields{
 					"error":      err,
 					"network_id": n.id,
 					"subnet":     s.subnetIP,
 				}).Warn("Failed to clean up iptables rules during overlay network deletion")
 			}
-			if err := programInput(s.vni, false); err != nil {
+			if err := d.programInput(s.vni, false); err != nil {
 				log.G(context.TODO()).WithFields(log.Fields{
 					"error":      err,
 					"network_id": n.id,
@@ -430,8 +430,11 @@ func (n *network) setupSubnetSandbox(s *subnet, brName, vxlanName string) error
 		return fmt.Errorf("bridge creation in sandbox failed for subnet %q: %v", s.subnetIP.String(), err)
 	}
 
-	err := createVxlan(vxlanName, s.vni, n.maxMTU())
+	v6transport, err := n.driver.isIPv6Transport()
 	if err != nil {
+		log.G(context.TODO()).WithError(err).Errorf("Assuming IPv4 transport; overlay network %s will not pass traffic if the Swarm data plane is IPv6.", n.id)
+	}
+	if err := createVxlan(vxlanName, s.vni, n.maxMTU(), v6transport); err != nil {
 		return err
 	}
 
@@ -522,12 +525,12 @@ func (n *network) initSubnetSandbox(s *subnet) error {
 	// Program iptables rules for mandatory encryption of the secure
 	// network, or clean up leftover rules for a stale secure network which
 	// was previously assigned the same VNI.
-	if err := programMangle(s.vni, n.secure); err != nil {
+	if err := n.driver.programMangle(s.vni, n.secure); err != nil {
 		return err
 	}
-	if err := programInput(s.vni, n.secure); err != nil {
+	if err := n.driver.programInput(s.vni, n.secure); err != nil {
 		if n.secure {
-			return multierror.Append(err, programMangle(s.vni, false))
+			return multierror.Append(err, n.driver.programMangle(s.vni, false))
 		}
 	}
 
diff --git a/libnetwork/drivers/overlay/ov_utils.go b/libnetwork/drivers/overlay/ov_utils.go
index 46b61b6dee159..1ef14ecfc2222 100644
--- a/libnetwork/drivers/overlay/ov_utils.go
+++ b/libnetwork/drivers/overlay/ov_utils.go
@@ -5,6 +5,7 @@ package overlay
 import (
 	"context"
 	"fmt"
+	"net"
 	"syscall"
 
 	"github.com/containerd/log"
@@ -56,7 +57,7 @@ func createVethPair() (string, string, error) {
 	return name1, name2, nil
 }
 
-func createVxlan(name string, vni uint32, mtu int) error {
+func createVxlan(name string, vni uint32, mtu int, vtepIPv6 bool) error {
 	vxlan := &netlink.Vxlan{
 		LinkAttrs: netlink.LinkAttrs{Name: name, MTU: mtu},
 		VxlanId:   int(vni),
@@ -67,6 +68,19 @@ func createVxlan(name string, vni uint32, mtu int) error {
 		L2miss:    true,
 	}
 
+	// The kernel restricts the destination VTEP (virtual tunnel endpoint) in
+	// VXLAN forwarding database entries to a single address family, defaulting
+	// to IPv4 unless either an IPv6 group or default remote destination address
+	// is configured when the VXLAN link is created.
+	//
+	// Set up the VXLAN link for IPv6 destination addresses by setting the VXLAN
+	// group address to the IPv6 unspecified address, like iproute2.
+	// https://github.com/iproute2/iproute2/commit/97d564b90ccb1e4a3c756d9caae161f55b2b63a2
+	// https://patchwork.ozlabs.org/project/netdev/patch/20180917171325.GA2660@localhost.localdomain/
+	if vtepIPv6 {
+		vxlan.Group = net.IPv6unspecified
+	}
+
 	if err := ns.NlHandle().LinkAdd(vxlan); err != nil {
 		return fmt.Errorf("error creating vxlan interface: %v", err)
 	}
diff --git a/libnetwork/drivers/overlay/overlay.go b/libnetwork/drivers/overlay/overlay.go
index 93082422e2269..d320c1602a1a0 100644
--- a/libnetwork/drivers/overlay/overlay.go
+++ b/libnetwork/drivers/overlay/overlay.go
@@ -7,6 +7,7 @@ package overlay
 import (
 	"context"
 	"fmt"
+	"net"
 	"sync"
 
 	"github.com/containerd/log"
@@ -27,16 +28,16 @@ const (
 var _ discoverapi.Discover = (*driver)(nil)
 
 type driver struct {
-	bindAddress      string
-	advertiseAddress string
-	config           map[string]interface{}
-	peerDb           peerNetworkMap
-	secMap           *encrMap
-	networks         networkTable
-	initOS           sync.Once
-	localJoinOnce    sync.Once
-	keys             []*key
-	peerOpMu         sync.Mutex
+	bindAddress, advertiseAddress net.IP
+
+	config        map[string]interface{}
+	peerDb        peerNetworkMap
+	secMap        *encrMap
+	networks      networkTable
+	initOS        sync.Once
+	localJoinOnce sync.Once
+	keys          []*key
+	peerOpMu      sync.Mutex
 	sync.Mutex
 }
 
@@ -71,11 +72,27 @@ func (d *driver) IsBuiltIn() bool {
 	return true
 }
 
-func (d *driver) nodeJoin(advertiseAddress, bindAddress string, self bool) {
-	if self {
+// isIPv6Transport reports whether the outer Layer-3 transport for VXLAN datagrams is IPv6.
+func (d *driver) isIPv6Transport() (bool, error) {
+	// Infer whether remote peers' virtual tunnel endpoints will be IPv4 or IPv6
+	// from the address family of our own advertise address. This is a
+	// reasonable inference to make as Linux VXLAN links do not support
+	// mixed-address-family remote peers.
+	if d.advertiseAddress == nil {
+		return false, fmt.Errorf("overlay: cannot determine address family of transport: the local data-plane address is not currently known")
+	}
+	return d.advertiseAddress.To4() == nil, nil
+}
+
+func (d *driver) nodeJoin(data discoverapi.NodeDiscoveryData) error {
+	if data.Self {
+		advAddr, bindAddr := net.ParseIP(data.Address), net.ParseIP(data.BindAddress)
+		if advAddr == nil {
+			return fmt.Errorf("invalid discovery data")
+		}
 		d.Lock()
-		d.advertiseAddress = advertiseAddress
-		d.bindAddress = bindAddress
+		d.advertiseAddress = advAddr
+		d.bindAddress = bindAddr
 		d.Unlock()
 
 		// If containers are already running on this network update the
@@ -84,6 +101,7 @@ func (d *driver) nodeJoin(advertiseAddress, bindAddress string, self bool) {
 			d.peerDBUpdateSelf()
 		})
 	}
+	return nil
 }
 
 // DiscoverNew is a notification for a new discovery event, such as a new node joining a cluster
@@ -91,10 +109,10 @@ func (d *driver) DiscoverNew(dType discoverapi.DiscoveryType, data interface{})
 	switch dType {
 	case discoverapi.NodeDiscovery:
 		nodeData, ok := data.(discoverapi.NodeDiscoveryData)
-		if !ok || nodeData.Address == "" {
-			return fmt.Errorf("invalid discovery data")
+		if !ok {
+			return fmt.Errorf("invalid discovery data type: %T", data)
 		}
-		d.nodeJoin(nodeData.Address, nodeData.BindAddress, nodeData.Self)
+		return d.nodeJoin(nodeData)
 	case discoverapi.EncryptionKeysConfig:
 		encrData, ok := data.(discoverapi.DriverEncryptionConfig)
 		if !ok {
diff --git a/libnetwork/drivers/overlay/peerdb.go b/libnetwork/drivers/overlay/peerdb.go
index c636201e2432e..35e7b2e6713d8 100644
--- a/libnetwork/drivers/overlay/peerdb.go
+++ b/libnetwork/drivers/overlay/peerdb.go
@@ -429,7 +429,7 @@ func (d *driver) peerFlushOp(nid string) error {
 func (d *driver) peerDBUpdateSelf() {
 	d.peerDbWalk(func(nid string, pkey *peerKey, pEntry *peerEntry) bool {
 		if pEntry.isLocal {
-			pEntry.vtep = net.ParseIP(d.advertiseAddress)
+			pEntry.vtep = d.advertiseAddress
 		}
 		return false
 	})


Test patch: