dotcloud/docker — pull request #6810 by phemmer (2014-07-02 04:15:19+00:00)
Title: Support hairpin NAT

This re-applies commit b39d02b with additional iptables rules to solve the issue with containers routing back into themselves.

The previous issue with this attempt was that the DNAT rule would send traffic back into the container it came from. When this happens you have 2 issues:

1. reverse path filtering. The container is going to see the traffic coming in from the outside and it's going to have a source address of itself. So reverse path filtering will kick in and drop the packet.
2. direct return mismatch. Assuming you turned reverse path filtering off, when the packet comes back in, it's goign to have a source address of itself, thus when the reply traffic is sent, it's also going to have a source address of itself. But the original packet was sent to the docker host IP address, so the traffic will be dropped because it's coming from an address which the original traffic was not sent to (and likely with an incorrect port as well).

The solution to this is to masquerade the traffic when it gets routed back into the origin container. However for this to work you need to enable hairpin mode on the bridge port, otherwise the kernel will just drop the traffic.
The hairpin mode set is part of libcontainer, while the MASQ change is part of docker.

The libcontainer change is docker/libcontainer#62

----

Note, with this change, it is *almost* possible to remove the docker proxy. The only thing left that the proxy handles is connecting to a port mapping via localhost from the docker server (the [TestAllocateTCPPortLocalhost](https://github.com/dotcloud/docker/blob/v1.0.1/integration/runtime_test.go#L506) test).
This can easily be supported by iptables, but requires 3 changes:

1. The `! -d 127.0.0.0/8` on the [`-t nat OUTPUT` rule](https://github.com/dotcloud/docker/blob/v1.0.1/pkg/iptables/iptables.go#L49) needs to be removed. However from the comment on [line 131](https://github.com/dotcloud/docker/blob/v1.0.1/pkg/iptables/iptables.go#L131), it looks like this was deliberately added for some reason.
2. A `MASQUERADE` rule needs to be added to masquerade traffic so it's not appearing to come from `127.0.0.1`.
2. `net.ipv4.conf.$iface.route_localnet=1` needs to be set on the bridge interface. While `route_localnet` is there for security reasons (since local network traffic should never be routed), the `route_localnet` inside a container's namespace will prevent any container from being able to send localnet traffic to another through the bridge. And then the `accept_local` setting in the container will prevent the traffic from being received as well.

I ran the integration test suite with the proxy disabled, and the only tests that failed were those localhost tests.