Falling deep down a networking rabbit hole, one hop at a time
By: Alex Boten
Tags: Networking, Virtualbox, Network, Debugging, Vagrant

I ran into an interesting networking problem last week while configuring a multi-host Vagrant environment. I couldn’t find a lot of information about it anywhere so I thought, why not share it? Maybe it will save time for the next poor soul who tries to get VirtualBox virtual machines talking over MacVTap interfaces. Mostly I’m just hoping that I’m not alone out there trying to make this work.

One of my all time favorites https://xkcd.com/979/

Getting started

First things first, let’s launch a couple of hosts. This is super easy to do using Vagrant. I created two VirtualBox instances using standard Ubuntu distributions via the following configuration. I should add that they are connected on a private network.

Once the hosts come up, I run a simple ping to make sure everything works. Looking good so far:

$ vagrant up

Bringing machine 'host1' up with 'virtualbox' provider...

Bringing machine 'host2' up with 'virtualbox' provider...

==> host1: Importing base box 'ubuntu/bionic64'...

... $ vagrant ssh host1

vagrant@host1:~$ ip a

....

3: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000

link/ether 08:00:27:65:8e:21 brd ff:ff:ff:ff:ff:ff

inet 172.28.128.3/24 brd 172.28.128.255 scope global dynamic $ vagrant ssh host2

vagrant@host2:~$ ping 172.28.128.3

PING 172.28.128.3 (172.28.128.3) 56(84) bytes of data.

64 bytes from 172.28.128.3: icmp_seq=1 ttl=64 time=0.711 ms

64 bytes from 172.28.128.3: icmp_seq=2 ttl=64 time=0.487 ms

Network Namespaces + MacVTAP

Here’s where we get to the fun stuff. I wanted to use network namespaces for this project. If you’re not familiar with network namespaces, they’re a mechanism to provide a layer of isolation for the network stack in Linux. In general, many people operate in the default namespace without ever thinking about it. If you’ve ever used Docker networks, there’s a good chance you’ve used multiple network namespaces without even knowing. Creating a namespace can be done in a single command:

root@host2:~# ip netns add winter-sparkles

root@host2:~# ip netns list

winter-sparkles

You can then hop into your namespace and poke around:

root@host2:~# ip netns exec winter-sparkles bash

root@host2:~# ip a

1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN group default qlen 1000

link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

root@host2:~# route

Kernel IP routing table

Destination Gateway Genmask Flags Metric Ref Use Iface

root@host2:~# exit

MacVTap is a Linux device driver that allows us to create virtual interfaces with their own IP and physical addresses. With it, we’re able to create as many virtual interfaces as we need in order to provide each network namespace with its own interface. There are other mechanisms that can accomplish this (queue acronym soup), but MacVTap is what I was working with. Creating a MacVTap interface looks like:

### back in the default network namespace

root@host2:~# ip a

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000

link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

inet 127.0.0.1/8 scope host lo

valid_lft forever preferred_lft forever

inet6 ::1/128 scope host

valid_lft forever preferred_lft forever

2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000

link/ether 02:62:a9:91:21:9f brd ff:ff:ff:ff:ff:ff

inet 10.0.2.15/24 brd 10.0.2.255 scope global dynamic enp0s3

valid_lft 85922sec preferred_lft 85922sec

inet6 fe80::62:a9ff:fe91:219f/64 scope link

valid_lft forever preferred_lft forever

3: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000

link/ether 08:00:27:c1:70:10 brd ff:ff:ff:ff:ff:ff

inet 172.28.128.4/24 brd 172.28.128.255 scope global dynamic enp0s8

valid_lft 722sec preferred_lft 722sec

inet6 fe80::a00:27ff:fec1:7010/64 scope link

valid_lft forever preferred_lft forever

root@host2:~# ip link add link enp0s8 name enp0s8vtap type macvtap mode bridge

root@host2:~# ip a

...

4: enp0s8vtap@enp0s8: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 500

link/ether 92:f9:94:a2:99:8c brd ff:ff:ff:ff:ff:ff

I’ll then take the interface and add it to the namespace I created earlier. To make the interface useful, we’ll need to give it an IP address:

root@host2:~# ip link set enp0s8vtap netns winter-sparkles up root@host2:~# ip netns exec winter-sparkles bash

root@host2:~# ip link set dev lo up

root@host2:~# ip address add 172.28.128.104/24 dev enp0s8vtap #### back on host1

root@host1:~# ping 172.28.128.104

And that’s when the problems started. On a physical network, the following would work:

connect two hosts to a router

create a new interface on one of those hosts

assign it an address

ping to it

jump for joy as happiness ensues

Here though, complete silence. Time to crack open tcpdump :

#### host1

root@host1:~# tcpdump -nl -i any icmp &

root@host1:~# ping 172.28.128.104

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode

listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes

04:46:21.727877 IP host1 > 172.28.128.104: ICMP echo request, id 2108, seq 39, length 64

04:46:22.778410 IP host1 > 172.28.128.104: ICMP echo request, id 2108, seq 40, length 64 #### host2 inside default namespace

root@host2:~# tcpdump -nl -i any icmp #### host2 inside winter-sparkles namespace

root@host2:~# tcpdump -nl -i any icmp

Nothing. The pings never make it to the second host, either inside or outside of the namespace. Let’s widen the search and learn.

Looking for martians

Ok, bare with me here. If your reaction to this heading is to think I’ve gone insane jumping straight to the theory that aliens have abducted my packets, you may be where I was when I first learned about martian packets. When a host receives packets on an interface and the source for the packet is routed on a different interface, it logs it as a martian or silently drops the packet. By default logging for martians is off in my machines, so I turned it on using sysctl:

NOTE: sysctl will only apply within their namespace, you’ll need to apply them in both namespaces

#### host2 in both default & winter-sparkles

root@host2:~# sysctl -a | grep martians

net.ipv4.conf.all.log_martians = 0

net.ipv4.conf.default.log_martians = 0

root@host2:~# sysctl net.ipv4.conf.all.log_martians=1

root@host2:~# sysctl net.ipv4.conf.default.log_martians=1

root@host2:~# tail -f /var/log/kern.log | grep martian

Nope, not that either, exhausted another theory. Onto the next hop.

Address Resolution Protocol

Running tcpdump with a host filter revealed some interesting details. I could see the Address Resolution Protocol (ARP) requests from host1 and the responses as well. Looking at the ARP table, it’s clear host1 was getting some traffic to the MacVTap interface.

#### host1

root@host1:~# tcpdump -nl -i any host 172.28.128.104 &

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode

listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes

root@host1:~# ping 172.28.128.104

05:52:22.713348 ARP, Reply 172.28.128.104 is-at 66:20:3e:d8:81:b6, length 46

Looking through the documentation for internal network configuration in VirtualBox didn’t give me a lot to go on. I started poking around the user interface. I knew I was really grasping at straws but I was running out of options I could think of. I found the “Promiscuous Mode” option and set it from “Deny” to “Allow”, still no changes.

Finally

After getting some more sleep and coming back at it with fresh eyes and coffee, I thought of something else to try. Something I read in the VirtualBox docs make me think of it:

Once there is more than one active virtual network card with the same internal network ID, the Oracle VM VirtualBox support driver will automatically wire the cards and act as a network switch

Reading that description, I wondered if spoofing the Media Access Control (MAC) address of the device would allow my virtual interface to connect to the network. Spoofing means to act as something else, in my case, the MacVTap interface would pretend to be the original interface VirtualBox created.

#### in the default namespace

root@host2:~# ip link set enp0s8 down

root@host2:~# ip link set enp0s8 address 66:20:3e:d8:81:b6

root@host2:~# ip link set enp0s8 up #### in the winter-sparkles namespace

root@host2:~# ip link set enp0s8vtap down

root@host2:~# ip link set enp0s8vtap address 08:00:27:ae:cd:94

root@host2:~# ip link set enp0s8vtap up #### back on host1

root@host1:~# ping 172.28.128.104

PING 172.28.128.104 (172.28.128.104) 56(84) bytes of data.

64 bytes from 172.28.128.104: icmp_seq=1 ttl=64 time=0.593 ms

64 bytes from 172.28.128.104: icmp_seq=2 ttl=64 time=0.726 ms

At long last, there it is. This means that as far as I understand, I will not be able to use MacVTap interfaces if I want to use internal networks, as it appears there is some MAC filtering happening here. Days have gone by, but dammit it feels amazing to have gotten to the bottom of the missing packets. Of course, by this point I have no idea what I was originally trying to do. Would love to hear from others that may have gotten further along this route!