Repository: moby/moby (language: go)

Issue:
Failed to receive UDP traffic after container restart
I start a container and share a port for UDP traffic as this:

docker run —rm -p 5060:5060/udp —name host1 -i -t ubuntu:14.04

Then in that container I wait for traffic with:

nc -u -l 5060

I then generate traffic from another machine:

nc -u <docker_host_ip> 5060

Then everything works fine and I can see that I receive the UDP traffic in the container.

But when I exit the container and do the same thing again, then I can no longer receive UDP traffic in the docker container.
If I wait for about 5 minutes before I start to send it will work though. I have also noticed that if the sender change the port it is binding to locally it will also work.  So there seems to be some mapping that is not deleted when the docker container is removed.



Discussion / hints:
This issue is due to conntrack.  The linux kernel keeps state of each connection. Even though 
udp is connectionless, if you use

```
sudo cat /proc/net/ip_conntrack
```

you will see a lot of entries. The output shows that the container address is still the last one before restart,  the state also prevents packet form arriving at the new container, the reason is this:

> For a connection, the first packet will go through the iptables's NAT table, and that's where docker routes packet to its own chain, then to the right container.

When you restart the container, container's ip has changed, so the DNAT rule, which will route to the new address. But the old connection's state in conntrack is not cleared. So when a packet arrives, it will not go through NAT table again, because it is not "the first" packet. So the solution is clearing the conntrack, which can be done as follows:

```
sudo conntrack -D -p udp
```

(you will need `sudo apt-get install conntrack`)

Looking forward to Docker's solution.

Same problem on my side (openvpn within a container). I could resolve it temporary with

iptables --table raw --append PREROUTING --protocol udp --source-port 4000 --destination-port 4000 --jump NOTRACK

Running on docker host. It's ugly but gets the job done.

IMHO the correct solution would be to clean up to conntrack-table after adjusting iptables.

Definitely looking forward to a fix for this one.

Seems like working for me with 3.18.0 kernel.

The UDP proxy has always had issues with packet loss, we've never found 
a good answer for it.

-Erik

On Mon, Jan 5, 2015 at 9:26 AM, Alexander Morozov 
notifications@github.com wrote:

> Seems like working for me with 3.18.0 kernel.
> 
> —
> Reply to this email directly or view it on GitHub.
> 
> # 

I'm using CentOS 6.6, kernel 2.6.32-504.1.3.el6.x86_64.  Seems like Docker should be responsible for (or at least facilitate through configuration) expiring conntrack table entries.  

I too would like to see some real solution to this.

Hi, we would also need to know when this issue makes progress, what the impediments to fix this bug? Can we help in any way with details? We run Consul and at some point (I guess after some restarts), the nodes start "suspecting each other" (per gossip protocol); the nodes can receive the udp that they are being suspected and they try to reply with hey i am alive, but the reply never reaches destination.

Is this a priority? is it hard to reproduce or debug? can we help with more concrete data?
i reproduced it with kernel 3.13

Flushing the conntrack table worked for me, but I'm running on a dev machine and not prod, I'll have to give @liyichao's answer a go if/when we hit this in prod.

Is there any reason why the conntrack entries can't just be removed when docker determines a container stopped?

@grimmy No, fix should be not too difficult to implement. After removing iptables-entries just call conntrack --delete with similar arguments (ip + port)

ok, that's what i figured.  I'll see if i can find some time to put a pull request together unless someone else wants to jump on it.

Hi,

I applied a patch in the cleanup callback of mapper.go, adding conntrack delete for the container ip as the source ip in 3 places in mapper.go, including Unmap, and cleanup functions. Did not succeed ie. serf gossip protocol which i run over udp complains that packages do not make across and blacklist other nodes in their memberlist. Either there must be other places to do this, or this should be also done on the remote nodes.

Normally this should be done via accessible "objects", but I have not found a suitable one, either in docker or as a golang import, and started by calling a command in the OS (which of course is not a portable solution, but one to check assumptions). 

```
cleanup := func() error {
    // need to undo the iptables rules before we return
    if m.userlandProxy != nil {
        m.userlandProxy.Stop()
    }
    pm.forward(iptables.Delete, m.proto, hostIP, allocatedHostPort, containerIP.String(), containerPort)
    if err := pm.Allocator.ReleasePort(hostIP, m.proto, allocatedHostPort); err != nil {
        return err
    }
    exec.Command("/usr/sbin/conntrack", "-D", "-s", containerIP.String()).Run()
    return exec.Command("/usr/sbin/conntrack", "-F").Run()
}
```

Hi, any feedback on my attempt to start a way to fix this?

I've been cheating locally when it happens by using "conntrack -F" next time it happens I'll try with just the specific ip address.

hi,

but i called -F too, probably in the wrong place.

for sure only the local tablea need to be flushed, not the remote ones
right?

În data de miercuri, 27 mai 2015, grimmy notifications@github.com a scris:

> I've been cheating locally when it happens by using "conntrack -F" next
> time it happens I'll try with just the specific ip address.
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/docker/docker/issues/8795#issuecomment-106026086.

I haven't had to do anything on the remote end.  But I do have multiple containers talking to each other and external devices over UDP.  The first time this happend (and I discovered that it was conntrack) was that there was a conntrack entry for an external device pointing to an old container.  Doing "conntrack -F" cleared that and then the next packet from that external device made it to the correct container.

So we're running StatsD in a Docker container on RHEL 7 and ran into this problem when the Docker service is restarted, which in turn restarts Docker. The UDP packet to StatsD were arriving on interface but not making it through to the container and IPTables wasn't blocking it, which led us to this thread. 

The solution for us was to use conntrack to delete only the states for the things that are not working so that we have the least impact on existing states. In the SystemD unit file that launches the Docker container for StatsD, running a ExecStartPre with conntrack to delete the specific states that are UDP and 8125 has solved this problem for us. Running conntrack -F really seems a bit brute force for our requirements:

```
# grep -B1 run /etc/systemd/system/statsd.service 
ExecStartPre=/sbin/conntrack -D -p udp --orig-port-dst 8125
ExecStart=/usr/bin/docker run -p 8125:8125/udp -p 8126:8126 \
```

Yes the -F has only been preformed on dev workstations and of course not in prod.  This really just needs to be fixed in docker but @nmarasoiu hasn't had any success and I haven't had time to fix it either.

ping @mavenugo @mrjana is this Libnetwork territory now? Wondering if progress was made in this area

It's been several months w/o any posts.  Is this issue being tracked elsewhere?

I have a similar problem (UDP traffic from within container to world sometimes doesn't get routed back to container), however the workaround (conntrack -F) doesn't work. Note the work "sometimes" - it looks like it depends on destination IP, however I can't say for sure. The IPs I am using are in 10.0.0.0/8 range, so there is no collision there.

Workaround for me is using --net=host when running container, but would love a real solution and was hoping this issue would solve my problem too. Is anyone working on this?

hi

i recall this happens in container restart right?

is it possible to delete the container and relaunch a fresh one ? hope
image still cached in some local registry? or u have data in container?
using mounts may enable u statelss container cause they are best used for
short term tasks and erased after.. just an idea

În Mi, 9 sept. 2015 la 08:34 andyka notifications@github.com a scris:

> I have a similar problem (UDP traffic from within container to world
> sometimes doesn't get routed back to container), however the workaround
> (conntrack -F) doesn't work. Note the work "sometimes" - it looks like it
> depends on destination IP, however I can't say for sure. The IPs I am using
> are in 10.0.0.0/8 range, so there is no collision there.
> 
> Workaround for me is using --net=host when running container, but would
> love a real solution. Is anyone working on this?
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/docker/docker/issues/8795#issuecomment-138790050.

On Sep 9, 2015, at 2:58 PM, nicu marasoiu notifications@github.com wrote:

> i recall this happens in container restart right?

No.  Happens when re-launching a container on the same host port, too.

exactly.

cant you avoid using same port, or not reusing container at all but
starting a new one from a local image store like the local docker images or
local registry?

În data de miercuri, 9 septembrie 2015, Brian Lalor <
notifications@github.com> a scris:

> On Sep 9, 2015, at 2:58 PM, nicu marasoiu <notifications@github.com
> <javascript:_e(%7B%7D,'cvml','notifications@github.com');>> wrote:
> 
> > i recall this happens in container restart right?
> 
> No. Happens when re-launching a container on the same host port, too.
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/docker/docker/issues/8795#issuecomment-139009616.

Love to see this fix too. Having problem with consul all the time now. For who searching for work around I used

```
docker run --net=host --privileged --rm cap10morgan/conntrack -D -p udp
docker run --net=host --privileged --rm cap10morgan/conntrack -F
```

Not sure what it does but it fixes the problem for me

@winggundamth for the conntrack fix do I need to stop currently running docker containers or can I run it while they are running?

@sstarcher No you don't have to stop container.

Than the docker UDP bug is not my problem.  We run 5 leaders on their own nodes in AWS using t2.micro's  they don't go above 10% CPU usage and the networking IN+OUT is around 3MB/s.  I'm bumping my nodes up to m3.mediums to remove any question about resource utilization on the nodes to see if the leader election issue happens again.  I recently spun up an independent cluster with no agents connected of 5 leader nodes and it did not have the leader election issue over a 12 hour period where our other 5 leaders had the issue 5 times within that 12 hour period.

I am also encountering this issue with Consul whenever the container is restarted.

Running this manually on the host clears the consul instability problem, but it's not an optimal solution:
`docker run --net=host --privileged --rm cap10morgan/conntrack -D -p udp`

:+1: 

I've seen the same issue occurring, using conntrack seems to resolve the issue as a workaround however docker should support this.

I am on CentOS 7 with Docker 1.9.0. I've got consul running in a container (1 server, 1 agent). After they startup and the agent connects, they immediately start with the "refuting a suspect message" and "suspect <host> has failed, no acks received". 

I've tried running (on the host directly): `conntrack -D p udp` and `conntrackt -F` with no change in output.

@mindscratch if you run `conntrack -F` and then restart the consul containers, things should work at that point.

I've tried that with no luck. I'm going to deploy consul on the hosts using
marathon.
On Nov 18, 2015 4:15 PM, "Erik Kristensen" notifications@github.com wrote:

> @mindscratch https://github.com/mindscratch if you run conntrack -F and
> then restart the consul containers, things should work at that point.
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/docker/docker/issues/8795#issuecomment-157865100.

Struggeling with this issue since more than a year now.
I'm using docker to compose an ELK environment and am collecting Firewall traffic logs via SYSLOG to Logstash at a rate of approx. 250 Events/sec.
My workaround is an explicit 'conntrack -D -p udp' in EVERY of my docker scripts, but everytime someone performs a manual "docker restart", my SYSLOG traffic is going to hell.
In my point of view it shouldn't be a big deal for the docker daemon to clear the relevant conntrack table entrys at a container stop or restart.
This bug is really annoying for me but i'm lacking of "coding skills" to implement a fix by myself :-(

Are you hitting conntrack limits? Should be very spammy in dmesg if you 
are. There is a set of sysctls (sysctl -A | grep conntrack) that allow 
you to increase the connection throttles. Netfilter kind of sucks at 
this, and the throttle is nearly impossible to remove.

/cc @discordianfish

-Erik

On 8 Jan 2016, at 5:25, Andreas Schäfer wrote:

> Struggeling with this issue since more than a year now.
> I'm using docker to compose an ELK environment and am collecting 
> Firewall traffic logs via SYSLOG to Logstash at a rate of approx. 250 
> Events/sec.
> My workaround is an explicit 'conntrack -D -p udp' in EVERY of my 
> docker scripts, but everytime someone performs a manual "docker 
> restart", my SYSLOG traffic is going to hell.
> In my point of view it shouldn't be a big deal for the docker daemon 
> to clear the relevant conntrack table entrys at a container stop or 
> restart.
> This bug is really annoying for me but i'm lacking of "coding skills" 
> to implement a fix by myself :-(
> 
> ---
> 
> Reply to this email directly or view it on GitHub:
> https://github.com/docker/docker/issues/8795#issuecomment-169809348

:see_no_evil: :hear_no_evil: :speak_no_evil:

@andsch If you have an docker container that clears the conntrack table, you can apply an restart policy using docker to help clear the table upon an docker daemon restart.

These are all hacks though. Do we actually know the underlying cause? This is a super annoying bug for anyone actually using UDP.  I would love to work with a docker dev to get this fixed. It seems like this should be a high priority bug, but doesn't seem to be getting a lot of attention.

+1 for what @ekristen said.  This bug is a major nuisance for UDP users.

@mpeterss @CVTJNII @ekristen could someone give us a consistent (& simple) way to reproduce this issue so that we can dig into the root-cause and possible solution ? 
I couldn't reproduce the issue as mentioned in the issue description (yes it is 1 yr+ old :( ) in the latest (from master).

@erikh no, dmesg output is clean of conntrack issues. I don't think that there is a tuneable sysctl flag that solves this issue due to a high rate UDP stream.

@mavenugo it's quite simple to reproduce. i did is by using 'nping' to send traffic to syslog port udp/514 with a rate of one packet per second.

start a container with an exposed UDP Port, in my case it is syslog:
 docker run -d -p 514:514/udp logstash:0.1

i use "nping" to generate UDP traffic.
 nping --udp -p 514 -c 0 10.240.255.15

 iptables -L
[...]
ACCEPT     udp  --  anywhere             172.17.0.17          udp dpt:syslog

conntrack -L | grep 172.17.0.17 displays the session

 conntrack -L | grep 172.17.0.17
udp      17 29 src=192.168.0.1 dst=10.240.255.15 sport=1028 dport=514 [UNREPLIED] src=172.17.0.17 dst=192.168.0.1 sport=514 dport=1028 mark=0 use=1
[...]

after performing a "docker restart" on the "logstash" container a new bridge interface is spawned with a new IP address. iptables -L shows that the NAT destination IP for SYSLOG is increased by 1.

 iptables -L
[...]
ACCEPT     udp  --  anywhere             172.17.0.18          udp dpt:syslog

conntrack -L | grep 172.17.0.17 still produces the same unchanged output

 conntrack -L | grep 172.17.0.17
udp      17 29 src=192.168.0.1 dst=10.240.255.15 sport=1028 dport=514 [UNREPLIED] src=172.17.0.17 dst=192.168.0.1 sport=514 dport=1028 mark=0 use=1
[...]

there is NO conntrack session sourced by IP 192.168.0.1 with correct NAT destination of 172.17.0.18
after cleaning conntrackd a new session is allocated with correct destination IP of 172.17.0.18.

i totally aggree to @ekristen that this should be a mayor issue. 

@andsch @ekristen @CVTJNII docker 1.7 release & later brought in a number of docker networking enhancements & especially in the area of IP address management.
- a container restart (even stop/start sequence) will try to reuse the same ip-address (unless there is no other container steals that ip-address during the container restart process). When the IP is retained, the contracker cache issue reported issue doesn't show up.
- Docker 1.9 supported user-defined networks, where user can create bridge networks and isolate the containers within that network. so the ip-address allocation is constrained within that network.
  Example : 

```
$ sudo docker network create --subnet=23.0.0.0/30 udpnet
2cceef8d3f1b5144152406df67573fd8f285a33ecaa4dc95adc081689046cb2f
$ sudo docker run  -p 5060:5060/udp -i -t --net=udpnet ubuntu:14.04
root@8472d16809a7:/# nc -u -l 5060
```

As you can see, I assigned a custom subnet `23.0.0.0/30` which assures that there can be only one container that can run in that network (By default we reserve 3 ip-addresses for every network  1 for network-address, 1 for broadcast-address and 1 for subnet Gateway. Hence for /30, among 4 addresses, only 1 is available for a container). With that this container is always guaranteed to get the same ip-address `23.0.0.2`. With port-mapping, the udp client shouldnt care about these details and will be able to reach the udp server using the mapped port on the host.  Hence the application wont have any problems posed by contracker caching.
- in the upcoming release (docker 1.10), docker supports user-assignable ip-address for containers connected to user-defined networks. For example : 

```
$ sudo docker network create --subnet=23.0.0.0/16 --ip-range=23.0.0.0/24 udpnet
2cceef8d3f1b5144152406df67573fd8f285a33ecaa4dc95adc081689046cb2f
$ sudo docker run  -p 5060:5060/udp -i -t --net=udpnet --ip=23.0.1.2 ubuntu:14.04
root@8472d16809a7:/# nc -u -l 5060
```

the `--ip` param in `docker run` will guarantee that the container across restarts will get the same ip-address. Again this will take care of this issue.

@nmarasoiu @winggundamth am painfully aware of the consul and ip-address issue and the above suggestions will help as well.

Please let us know if any of the above suggestions doesn't work for your case.

Thanks to @mavenugo @shykes and whoever else took notice to this to get some answers, really appreciate it.

Thanks for the detailed response! I'm excited for the new changes coming with docker networking, however I have a few issues.

I've most definitely experience these problems the most with consul, even when the IP hasn't changed.
1. I don't think it is reasonable to say that if you are using udp and need reliable connections that you are going to have to implement a user defined network and assign an IP to your container so it never changes, this is bad UX. UDP and TCP are the backbone of networking, both need to be supported equally in a reliable manor.
2. I'd like to see that once a container is created and an IP address is assigned that it cannot be reassigned for the lifetime of the container, that means if I stop the original container it should still not be possible for another container to pick up the IP.

Hi, just wanted to chip in and mention that I have similar flapping issues related to consul / docker
(not due to restarts however)

My setup:

```
docker exec -ti f9457379e54d /consul/consul  version
Consul v0.6.3
Consul Protocol: 3 (Understands back to: 1)
```

Consul cluster of 3 running inside docker containers.
3 Separate VM's (vmware) on same Vlan (no firewalls at all)

First I make sure that I've run `conntrack -F or ip -s -s neigh flush all`
to flush the cache. I also rm any {raft,serf} files I might have in /var/tmp/consul

Cluster is started fresh using : 

node1 : 10.7.7.75
node2 : 10.7.7.79
node3 : 10.7.7.83

(the above ip addresses are changed for this post) (tried to be consistent).

```
#node 1, node 2 and node 3 are all started with the same command
#the ip addresses are changed for each node.

/usr/bin/docker run -d --privileged --net=host -v /var/tmp/consul:/data \
    -p 10.7.7.75:8300:8300/tcp \
    -p 10.7.7.75:8300:8300/udp \
    -p 10.7.7.75:8301:8301/tcp \
    -p 10.7.7.75:8301:8301/udp \
    -p 10.7.7.75:8302:8302/tcp \
    -p 10.7.7.75:8302:8302/udp \
    -p 10.7.7.75:8400:8400/tcp \
    -p 10.7.7.75:8400:8400/udp \
    -p 10.7.7.75:8500:8500/tcp \
    -p 10.7.7.75:8500:8500/udp \
    -p 10.7.7.75:8600:8600/tcp \
    -p 10.7.7.75:8600:8600/udp \
    gliderlabs/consul-agent -server -bind 10.7.7.75 -bootstrap-expect 3 -ui -dc A -config-file /data/server.json
```

```
cat /var/tmp/consul/server.json
{
  "disable_update_check": true,
  "dns_config": {
    "allow_stale": true,
    "node_ttl": "2s",
    "max_stale": "5s",
    "service_ttl": {
      "*": "5s",
      "api": "10s"
    }
  }
}
```

then I run join from one of the nodes.

```
docker exec -ti $(docker ps -lq) /bin/consul join 10.7.7.75 10.7.7.79 10.7.7.83
```

So far so good.

```
#node1 logs
2016/04/02 22:16:23 [ERR] agent: failed to sync remote state: No cluster leader
2016/04/02 22:16:38 [ERR] agent: coordinate update error: No cluster leader
2016/04/02 22:16:41 [INFO] serf: EventMemberJoin: node3 10.7.7.83
2016/04/02 22:16:41 [INFO] consul: adding LAN server node3 (Addr: 10.7.7.83:8300) (DC: A)
2016/04/02 22:16:41 [INFO] serf: EventMemberJoin: node2 10.7.7.79
2016/04/02 22:16:41 [INFO] consul: adding LAN server node2 (Addr: 10.7.7.79:8300) (DC: A)
2016/04/02 22:16:41 [INFO] consul: New leader elected: node2
2016/04/02 22:16:43 [INFO] agent: Synced service 'consul'
2016/04/02 22:18:07 [INFO] agent.rpc: Accepted client: 127.0.0.1:56493
2016/04/02 22:18:12 [INFO] agent.rpc: Accepted client: 127.0.0.1:56494

#node2 logs (leader)

2016/04/02 22:16:41 [INFO] consul: adding LAN server node3 (Addr: 10.7.7.83:8300) (DC: A)
2016/04/02 22:16:41 [WARN] raft: Heartbeat timeout reached, starting election
2016/04/02 22:16:41 [INFO] raft: Node at 10.7.7.79:8300 [Candidate] entering Candidate state
2016/04/02 22:16:41 [INFO] raft: Election won. Tally: 2
2016/04/02 22:16:41 [INFO] raft: Node at 10.7.7.79:8300 [Leader] entering Leader state
2016/04/02 22:16:41 [INFO] consul: cluster leadership acquired
2016/04/02 22:16:41 [INFO] consul: New leader elected: node2
2016/04/02 22:16:41 [INFO] raft: pipelining replication to peer 10.7.7.83:8300
2016/04/02 22:16:41 [INFO] raft: pipelining replication to peer 10.7.7.75:8300
2016/04/02 22:16:41 [WARN] raft: Remote peer 10.7.7.75:8300 does not have local node 10.7.7.79:8300 as a peer
2016/04/02 22:16:41 [INFO] consul: member 'node2' joined, marking health alive
2016/04/02 22:16:41 [INFO] consul: member 'node1' joined, marking health alive
2016/04/02 22:16:41 [INFO] consul: member 'node3' joined, marking health alive
2016/04/02 22:16:43 [INFO] agent: Synced service 'consul'

#node3 logs
2016/04/02 22:16:41 [INFO] agent: (LAN) joining: [10.7.7.75 10.7.7.79 10.7.7.83]
2016/04/02 22:16:41 [INFO] serf: EventMemberJoin: node1 10.7.7.75
2016/04/02 22:16:41 [INFO] consul: adding LAN server node1 (Addr: 10.7.7.75:8300) (DC: A)
2016/04/02 22:16:41 [INFO] serf: EventMemberJoin: node2 10.7.7.79
2016/04/02 22:16:41 [INFO] consul: adding LAN server node2 (Addr: 10.7.7.79:8300) (DC: A)
2016/04/02 22:16:41 [INFO] consul: Attempting bootstrap with nodes: [10.7.7.83:8300 10.7.7.75:8300 10.7.7.79:8300]
2016/04/02 22:16:41 [INFO] agent: (LAN) joined: 3 Err: <nil>
2016/04/02 22:16:41 [INFO] consul: New leader elected: node2
2016/04/02 22:16:41 [INFO] agent: Synced service 'consul'
2016/04/02 22:16:57 [INFO] agent.rpc: Accepted client: 127.0.0.1:34524
```

Then I start up another container on for example node1.

```
[root@node1 ~]# docker run -d -e CONSUL_SERVER=10.7.7.75 -e DATA_CENTER=A myregistry:5000/my-application:latest

f9457379e54d1262b0a4ce8b583b3e029f91c5cd8ac44d6f3468345a52cd5508
```

This particular container has a consul agent inside of it which starts up using supervisord as pid1 : 

```
# from the supervisord conf (the environment variables are replaced with  the -e variables I provided at the docker run command shown above.

/consul/consul agent -dc=%(ENV_DATA_CENTER)s -retry-join=%(ENV_CONSUL_SERVER)s -config-dir=/consul/config -pid-file=/consul/consul.pid -data-dir=/consul/data
```

The logs from consul monitor on node1 says:

```
2016/04/02 22:25:50 [INFO] serf: EventMemberJoin: f9457379e54d 172.17.0.3
2016/04/02 22:27:05 [INFO] serf: EventMemberFailed: f9457379e54d 172.17.0.3
2016/04/02 22:27:19 [INFO] serf: EventMemberJoin: f9457379e54d 172.17.0.3
2016/04/02 22:27:30 [INFO] memberlist: Marking f9457379e54d as failed, suspect timeout reached
2016/04/02 22:27:30 [INFO] serf: EventMemberFailed: f9457379e54d 172.17.0.3
2016/04/02 22:27:49 [INFO] serf: EventMemberJoin: f9457379e54d 172.17.0.3
2016/04/02 22:28:01 [INFO] memberlist: Marking f9457379e54d as failed, suspect timeout reached
2016/04/02 22:28:01 [INFO] serf: EventMemberFailed: f9457379e54d 172.17.0.3
2016/04/02 22:28:04 [INFO] serf: attempting reconnect to f9457379e54d 172.17.0.3:8301
2016/04/02 22:28:04 [INFO] serf: EventMemberJoin: f9457379e54d 172.17.0.3
2016/04/02 22:30:01 [INFO] serf: EventMemberFailed: f9457379e54d 172.17.0.3
2016/04/02 22:30:19 [INFO] serf: EventMemberJoin: f9457379e54d 172.17.0.3
```

logs from the container f9457379e54d says:

(never mind the time stamp since I've created this ticket after the fact and scrolled through the logs and picked parts that were related)

```
2016/04/02 23:06:25 [WARN] memberlist: Refuting a suspect message (from: node3)
2016/04/02 23:07:05 [WARN] memberlist: Was able to reach node3 via TCP but not UDP, network may be misconfigured and not allowing bidirectional UDP
2016/04/02 23:07:08 [INFO] serf: EventMemberFailed: ca3877daf6c5 172.17.0.2
2016/04/02 23:07:19 [WARN] memberlist: Refuting a suspect message (from: f9457379e54d)
2016/04/02 23:07:20 [INFO] serf: attempting reconnect to ca3877daf6c5 172.17.0.2:8301
2016/04/02 23:07:20 [INFO] serf: EventMemberJoin: ca3877daf6c5 172.17.0.2
2016/04/02 23:07:47 [INFO] agent.rpc: Accepted client: 127.0.0.1:37013
```

brought up another container with the same run commands as the previous container.

(ca3877daf6c5) on node1

Same flapping issues - serf checks going green and orange.

logs from node 3

```
2016/04/02 23:11:11 [ERR] memberlist: Push/Pull with f9457379e54d failed: dial tcp 172.7.0.3:8301: getsockopt: no route to host
2016/04/02 23:11:15 [INFO] serf: EventMemberFailed: ca3877daf6c5 172.17.0.2
2016/04/02 23:11:21 [INFO] serf: EventMemberJoin: ca3877daf6c5 172.17.0.2
2016/04/02 23:11:41 [INFO] memberlist: Suspect ca3877daf6c5 has failed, no acks received

2016/04/02 23:11:42 [ERR] memberlist: Push/Pull with f9457379e54d failed: dial tcp 172.17.0.3:8301: getsockopt: no route to host  ### <-- tried --expose 8301 when starting the application containers up however no luck - same issue and same messages. - clearly a routing issue however I can't understand why since there are no firewalls or similar in place.

2016/04/02 23:11:46 [INFO] memberlist: Marking ca3877daf6c5 as failed, suspect timeout reached
```

details of the members in the cluster.

```
[root@node1 ~]# docker exec -ti ca3877daf6c5 /consul/consul members -detailed
Node                                          Address            Status  Tags
ca3877daf6c5                                  172.17.0.2:8301    alive   build=0.6.3:c933efde,dc=sta,role=node,vsn=2,vsn_max=3,vsn_min=1
f9457379e54d                                  172.17.0.3:8301    alive   build=0.6.3:c933efde,dc=sta,role=node,vsn=2,vsn_max=3,vsn_min=1
node1  10.7.7.75:8301  alive   build=0.6.3:c933efde,dc=sta,expect=3,port=8300,role=consul,vsn=2,vsn_max=3,vsn_min=1
node2  10.7.7.79:8301  alive   build=0.6.3:c933efde,dc=sta,expect=3,port=8300,role=consul,vsn=2,vsn_max=3,vsn_min=1
node3  10.7.7.83:8301  alive   build=0.6.3:c933efde,dc=sta,expect=3,port=8300,role=consul,vsn=2,vsn_max=3,vsn_min=1
```

I've tried all possible combinations - exposing port 8301/udp and tcp on the application containers I start up .. running them with --net=host, --expose 8301/udp --expose 8301/tcp etc ..

always having the flapping issues.

@kamigerami it looks like your consul agents are advertising their docker bridge IP, meaning that the consul servers wouldn't be able to connect correctly, I suspect this is the case!

Should the conntack table suddenly flip to the wrong dst ip ? Is this a kernel bug?

Before = Before the flush of port 8125
After = After

I see something that doesnt seem right.
dst=172.17.0.7 

```
diff <( grep udp /tmp/before) <( grep upd /tmp/after ) |grep 172.17.0.7 |grep 8125
< udp      17 3 src=172.17.0.7 dst=10.8.128.122 sport=53587 dport=8125 [UNREPLIED] src=10.8.128.122 dst=10.8.147.207 sport=8125 dport=53587 mark=0 use=1
< udp      17 24 src=172.17.0.7 dst=10.8.128.122 sport=34143 dport=8125 [UNREPLIED] src=10.8.128.122 dst=172.17.0.7 sport=8125 dport=34143 mark=0 use=1
```

Comparing this to a healthy host

```
conntrack -L |grep 8125
udp      17 22 src=172.17.3.166 dst=10.8.128.122 sport=40613 dport=8125 [UNREPLIED] src=10.8.128.122 dst=10.8.147.207 sport=8125 dport=40613 mark=0 use=2
udp      17 29 src=172.17.0.7 dst=10.8.128.122 sport=53587 dport=8125 [UNREPLIED] src=10.8.128.122 dst=10.8.147.207 sport=8125 dport=53587 mark=0 use=1
udp      17 26 src=172.17.5.40 dst=10.8.128.122 sport=51816 dport=8125 [UNREPLIED] src=10.8.128.122 dst=10.8.147.207 sport=8125 dport=51816 mark=0 use=1
udp      17 29 src=172.17.0.7 dst=10.8.128.122 sport=34143 dport=8125 [UNREPLIED] src=10.8.128.122 dst=10.8.147.207 sport=8125 dport=34143 mark=0 use=1
udp      17 18 src=172.17.0.6 dst=10.8.128.122 sport=37371 dport=8125 [UNREPLIED] src=10.8.128.122 dst=10.8.147.207 sport=8125 dport=37371 mark=0 use=1
udp      17 19 src=172.17.0.6 dst=10.8.128.122 sport=44142 dport=8125 [UNREPLIED] src=10.8.128.122 dst=10.8.147.207 sport=8125 dport=44142 mark=0 use=1
udp      17 22 src=172.17.0.5 dst=10.8.128.122 sport=59844 dport=8125 [UNREPLIED] src=10.8.128.122 dst=10.8.147.207 sport=8125 dport=59844 mark=0 use=1
udp      17 21 src=172.17.5.226 dst=10.8.128.122 sport=50884 dport=8125 [UNREPLIED] src=10.8.128.122 dst=10.8.147.207 sport=8125 dport=50884 mark=0 use=1
udp      17 28 src=172.17.5.68 dst=10.188.132.40 sport=45125 dport=8125 [UNREPLIED] src=10.188.132.40 dst=10.8.147.207 sport=8125 dport=45125 mark=0 use=1
```

Edit: My problem may not 100% related to this.
I believe this will solve the problem for statsd. Statsd client library we use sends every 30 seconds. So when we get a bad entry they stay forever since the default udp timeout is 30 seconds.

The below will fix it for us:

```
echo "10" > /proc/sys/net/ipv4/netfilter/ip_conntrack_udp_timeout
```

I submitted a kernel bug, i think that's what this. https://bugzilla.kernel.org/show_bug.cgi?id=118951

@mavenugo new networking features are nice, but @ekristen is right. It's not a solution, it's just a workaround for some cases. We are using user-defined networks, but are constantly creating and deleting containers and also changing run parameters, so the only way to guaranty static ip is to manually set it, but it's hard to manage.

The real solution should be something like clearing conntrack table after container is stopped. Isn't just simply running `conntrack -D -g <container ip> -p udp` after container stops can fix the whole thing? imho it's critical problem for udp users with complex setup.

@stszap we will have to look into the details. Will get back to you on this.

Any updates? It seems that kuberneres resolved this problem. Maybe their solution should be used in docker?
ping @mavenugo @aboch @sanimej 
Can we use same stuff with conntrack as kubernetes?
@LK4D4 I will take a look..
Docker (and rkt) uses *iptables* target `MASQUERADE` in `nat`/`POSTROUTING`, but *iptables* works connection-oriented (as is TCP). To work around this *helper modules* for connection tracking had to be implemented (for example, for FTP). Without such *helper modules* UDP is tracked, too, but it sometimes quite does not work. For example, the assumption that the first packet establishes a session and return route fails for most UDP-protocols.

Absent a specialized module I suspect the following happens:

 * The first UDP datagram goes through `POSTROUTING`, because the virtual connection is in state `NEW`.
 * Subsequent packets therefore don't go through `POSTROUTING` anymore…
 * … and absent any further masquerading/`snat` the server emits UDP packets with the container's private address.
 * The router in your DC then drops packets with any "wrong" source address. Any IP that has not been assigned to your server.

It's hard to track this down if you use tools like **tcpdump** on *localhost*, or trying to reproduce this in a LAN where no filtering happens. If my assumption holds true you will be able to:

 * In a LAN, observe emitted UDP datagrams with a "wrong" (the container's) IP address as source.
 * In your DC, on a packet destination, log packets and don't find any or far less than sent by the corresponding source.
@sanimej Take a look at https://github.com/kubernetes/kubernetes/pull/32561
bump
I think I have this issue too. Took me a while to diagnose that UDP messages weren't delivered to container.  I see all UDP packets with `tcpdump` on the host, but in the container `tcpdump` shows only some of them. Had to switch my Graylog inputs to TCP.
@beatcracker we moved to our logging to JSON over TCP because Graylog would fragment and not be delivered because UDP.

This issue is a pain for me, because UDP is used a lot for SIP.
@grimmy On the second thought, this may be not a pure Docker issue, since I'm running Graylog in Rancher and it adds it's own networking layer. But anyway, lesson learned: UDP is PITA in the dockerized environments.
@beatcracker we will be fixing this is in Rancher soon.  Rancher extensively use UDP internally and thus a solution is essential.  We will be properly managing the conntrack tables.  Look for a solution in a 1.5.x release. 
FYI, @fcrisciani is currently working at a fix for this on docker side.
I assigned the issue to him then.
I ran into the same issues with docker swarm. I send gelf logs to logstash and after logstash is restarted logs are not reaching the logstash anymore. Still in need of a fix (instead of the proposed workarounds) that would support multiple instances of logstash and whenever a logstash container dies/is restarted no more udp packages should be sent to it (ip address cache removed)
@Hermain can you please post which is the docker version that you are using?
I am using version 17.03.1-ce.
@Hermain this fix is part of docker 17.05, so that's expected as 17.03.1 does not have this fix yet
Great I'll test it with 17.05.0 once its released.
Does this fix also apply to a swarm? Meaning when one node dies or a container dies and then a repica is created on a new node the packets will be forwarded to it?
@Hermain yep, should cover also swarm mode
I tested this on ubuntu with a one node swarm. I have logstash listening to gelf messages on localhost created by a container: 
``` 
docker run --log-driver=gelf --log-opt gelf-address=udp://127.0.0.1:12201 ubuntu /bin/sh -c 'while true; do date "+%d-%m-%Y %H:%M:%S:%3N" | xar printf "%s %s | 51c489da-2ba7-466e-abe1-14c236de54c5 | INFO | HostingLoggerExtensions.RequestFinished    | Request finished in 35.1624ms 200 application/json; charset=utf-8 message end\n"; sleep 1 ; done'
```
When I docker kill logstash while the log generator is running it takes ~ 5 minutes until the first udp messages reach logstash.

If I start the log generator after logstash everythings works perfectly right of the bat. To me it seems like the bug still exists.

I use Docker version 17.05.0-ce, build 89658be




I also tested it with docker version 17.05-ce and still have the issue. We are doing some tcpdump capture inside a container and all source IPs are modified by the docker interface IP
@Hermain tried the following:
log generator container: `docker run --log-driver=gelf --log-opt gelf-address=udp://127.0.0.1:5000 ubuntu /bin/sh -c 'while true; do date "+%d-%m-%Y %H:%M:%S:%3N" | xargs printf "%s %s | 51c489da-2ba7-466e-abe1-14c236de54c5 | INFO | HostingLoggerExtensions.RequestFinished    | Request finished in 35.1624ms 200 application/json; charset=utf-8 message end\n"; sleep 1 ; done'`

Daemon logs complains that there is destination for the logs:
```
ERRO[3031] Failed to log msg "" for logger gelf: gelf: cannot send GELF message: write udp 127.0.0.1:33019->127.0.0.1:5000: write: connection refused 
ERRO[3033] Failed to log msg "" for logger gelf: gelf: cannot send GELF message: write udp 127.0.0.1:33019->127.0.0.1:5000: write: connection refused 
ERRO[3035] Failed to log msg "" for logger gelf: gelf: cannot send GELF message: write udp 127.0.0.1:33019->127.0.0.1:5000: write: connection refused 
```

udp sync container (simple image that has networking tools): `docker run -d --name dst -p5000:5000/udp nicolaka/netshoot top`
run tcpdump: `docker exec -it dst tcpdump -eni eth0 udp and port 5000`

output like: 
```
01:03:45.183365 02:42:74:ca:af:79 > 02:42:ac:11:00:02, ethertype IPv4 (0x0800), length 560: 172.17.0.1.59017 > 172.17.0.2.5000: UDP, length 518
01:03:46.187920 02:42:74:ca:af:79 > 02:42:ac:11:00:02, ethertype IPv4 (0x0800), length 560: 172.17.0.1.49224 > 172.17.0.2.5000: UDP, length 518
01:03:47.193422 02:42:74:ca:af:79 > 02:42:ac:11:00:02, ethertype IPv4 (0x0800), length 561: 172.17.0.1.39521 > 172.17.0.2.5000: UDP, length 519
01:03:48.197281 02:42:74:ca:af:79 > 02:42:ac:11:00:02, ethertype IPv4 (0x0800), length 560: 172.17.0.1.43709 > 172.17.0.2.5000: UDP, length 518
```
Logs are coming correctly

Now the steps that you were mentioning:
`docker kill dst`
In the daemon logs (you have to set it in debug mode) I see the cleanup of the conntrack flows as expected from this patch: 
```
DEBU[3315] DeleteConntrackEntries purged ipv4:31, ipv6:0 
```
and again the daemon complains:
```
ERRO[3316] Failed to log msg "" for logger gelf: gelf: cannot send GELF message: write udp 127.0.0.1:33019->127.0.0.1:5000: write: connection refused 
ERRO[3318] Failed to log msg "" for logger gelf: gelf: cannot send GELF message: write udp 127.0.0.1:33019->127.0.0.1:5000: write: connection refused 
ERRO[3320] Failed to log msg "" for logger gelf: gelf: cannot send GELF message: write udp 127.0.0.1:33019->127.0.0.1:5000: write: connection refused 
ERRO[3322] Failed to log msg "" for logger gelf: gelf: cannot send GELF message: write udp 127.0.0.1:33019->127.0.0.1:5000: write: connection refused 
```

now: `docker start dst`
daemon stops posting errors
run again: `docker exec -it dst tcpdump -eni eth0 udp and port 5000`
and see the packets coming correctly:

```
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
01:09:11.202470 02:42:74:ca:af:79 > 02:42:ac:11:00:02, ethertype IPv4 (0x0800), length 560: 172.17.0.1.50807 > 172.17.0.2.5000: UDP, length 518
01:09:12.207231 02:42:74:ca:af:79 > 02:42:ac:11:00:02, ethertype IPv4 (0x0800), length 560: 172.17.0.1.47575 > 172.17.0.2.5000: UDP, length 518
```
@ggaugry when you use routing mesh the rewrite of the source IP address is expected to guarantee the return flow is working properly. Can you detail better what is the issue that you are referring to?
@fcrisciani any updates since you reproduced the issue? This bug still exists...
@Hermain Looking at the comment above, @fcrisciani was _not_ able to reproduce; udp traffic stopped when the `dst` container was killed, and started again when the `dst` container was started. Can you give more details? Exact steps to reproduce?
@Hermain the issue that you are experiencing is most likely the one fixed by this PR: https://github.com/docker/libnetwork/pull/1792 the 17.06-rc3 is out, you should try with that image to confirm that the issue is fixed.

That 5min delay to reconcile is exactly matching with the expiration time of the mac entry
The issue still persists. I now used the same tools as @fcrisciani to reproduce it again independently of logstash.

I have a one node docker swarm and use nicolaka/netshoot which I start with docker stack deploy and the following compose file:
```
version: "3.1"
services:
  udpReceiver:
    image: nicolaka/netshoot
    ports:
      - "127.0.0.1:12201:12201/udp"
    command: tcpdump -eni any udp and port 12201
```
If I now generate logs with docker run:
``` 
docker run --log-driver=gelf --log-opt gelf-address=udp://127.0.0.1:12201  ubuntu /bin/sh -c 'COUNTER=1;while true; do date "+%Y-%m-%d %H:%M:%S.%3N" | xargs printf "%s %s | 51c489da-2ba7-466e-abe1-14c236de54c5 | INFO | HostingLoggerExtensions.RequestFinished    | $COUNTER\n"; COUNTER=$((COUNTER+1)); sleep 1; done' 
```

and user docker logs on the netshoot container. I can see that it receives udp packages as expected. I leave the log generating container running and sending logs.

If I now ``` docker kill ${netStatContainerId} ``` docker swarm will spin up a new netstat and the logs from the log generator will not reach it. (I test this by doing docker logs on the new container and waiting for a minute --> Nothing happens)

If I stop the log generator and start a new one those logs reach the application.

Looks like this bug right?

Docker version returns:
```
Client:
 Version:      17.09.0-ce
 API version:  1.32
 Go version:   go1.8.3
 Git commit:   afdb6d4
 Built:        Tue Sep 26 22:42:18 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.09.0-ce
 API version:  1.32 (minimum version 1.12)
 Go version:   go1.8.3
 Git commit:   afdb6d4
 Built:        Tue Sep 26 22:40:56 2017
 OS/Arch:      linux/amd64
 Experimental: false
```
Or an even easier way to reproduce the problem:
Start the log generator from my last post, start the log receiver --> Logs will never reach the receiver as witnessed with ```docker service logs udpReceiver```.
With 17.05 at least it recovered after 5 minutes now it never recovers.
Any updates on this? We are seeing the same issue with web app containers sending udp messages to a rsyslog container. If the rsyslog container is killed and started again, the web app containers also need to be killed and started again for the udp messages to reach the rsyslog container. This is super weird. We are still on `17.05.0-ce, build 89658be`. Unlike with what @Hermain posted, it doesn't work even after 5 mins. Only a stop/start fixes this. 
I have the same issue with version `17.12.1-ce, build 3dfb8343b139d6342acfd9975d7f1068b5b1c3d3` running on AWS ECS. An outside host is sending UDP packets to the container every 500ms. Before I restart the container, everything is running fine. Once the container is restarted, I have around 50% of packet loss. When I have the issue, by using tcpdump inside the container, I see all the ingress and egress packets. However, when I run tcpdump on the docker host, I see half of the packet coming from the container.

The workaround `conntrack -F` is working for me. 
I see the same behaviour; the conntrack table is not flushed on a container restart so packets will be effectively blackholed until it is.

Only UDP ‘connections’ with the same source IP+port pair are affected, so not everyone will be hitting this bug.
I saw this behaviour on an instance that received ~4k syslog messages per second.
This didn't only happen after a restart though, the container would stop seeing traffic from some of the sources until I used conntrack to clear the table. Tcpdump on the host showed the traffic, but the container never saw it.

I've stopped using docker for that particular service since I can't afford to lose logs, and I couldn't find a solution.
@fcrisciani I understand you worked on this last year. Since this issue is currently closed but still present in 17.12.1, would it be better to have a new issue created? Is there any information you need to reproduce the issue? Let me know if I can help to get that fixed.
Since this issue is real, I'm attaching my least invasive solution I have found to work rather reliably, use ``nohup(1)`` or any other system dependent mechanism to keep this script running in the background and watch container starting up to clean ``conntrack`` entries corresponding to given container name that exposes given UDP port. Modify the script to adjust ``c`` and ``p`` appropriately.

```shell
#!/bin/bash

export PATH=/bin:/usr/bin:/sbin:/usr/sbin

# modify c and p to match your container name and UDP port
c=YOUR_CONTAINER_NAME
p=12345

docker events --filter type=container --filter event=start --filter container=$c | while read
do
    logger "$c restarted"
    conntrack -D -p udp --orig-port-dst $p 2>&1 >/dev/null
done
```

Since ``docker`` touches ``iptables`` on Linux host when containers are created, I do believe that it should also properly cleanup ``conntrack`` mappings belonging to the container.

@Hermain @mman looking into this, as far as I can see the problem seems to be in the state maintained inside the ipvs connection table. 
As for now if you enter in the ingress sandbox: `nsenter --net=/var/run/docker/netns/ingress_sbox` of each single node and you enable this knob: `echo 1 > /proc/sys/net/ipv4/vs/expire_nodest_conn`, this will allow ipvs to automatically purge the connection once the ipvs backend is not anymore available. With that the reproduction mentioned in https://github.com/moby/moby/issues/8795#issuecomment-334133087 seems to work properly. Still anyway working on making sure that that is enough
@fcrisciani should we reopen this issue, or is this a different cause as the original one and should we have a new issue for tracking?
@thaJeztah @fcrisciani Is the referenced issue docker/libnetwork#2154 fixing the UDP packet loss issue after container restart? 
@levesquejf also this one is needed: https://github.com/docker/libnetwork/pull/2243
I still see the same issue after 5 years since it was opened. Although, I am using the latest Docker version 19.03.5, in Docker Swarm and starting my services with docker stack. Container is 

It is exactly the same behavior as described in the initial post, when I first create stack it works. When I remove it and create again, containers do not receive traffic for exactly 5 minutes (300 seconds) and then they start to receive it. Deleting connections with conntrack at any point does not help at all. 

I have changes all parameters from `sysct` related to networking which had timeout or any parameter for 300 seconds to 30 seconds, but that did not changed behavior. 

Also, as mentioned in the first post, if I change the source port of the UDP sender, containers start to receive traffic. 

Containers are deployed over 3 docker swarm nodes with global mode. I don't have any specific configuration for network except exposing UDP ports where target and published ports are the same.

I don't believe that this issue is actually fixed. Any ideas what I could try more?
@dpajin I believe the issue is not fixed. I have successfully settled on the following workaround that seems to be stable for me for more than a year:

Somewhere in ``/etc/sysctl.conf`` I use value of 10 seconds for super aggressive timeouts, choose your own appropriate one.

```
net.netfilter.nf_conntrack_udp_timeout = 10
net.netfilter.nf_conntrack_udp_timeout_stream = 10
```
@mman, thank you for your reply and suggestion. Strangely, your change does not have any effect on the behavior in my setup. I will investigate this a bit more and eventually open a new issue.
> @mman, thank you for your reply and suggestion. Strangely, your change does not have any effect on the behavior in my setup. I will investigate this a bit more and eventually open a new issue.

Not working for me either. Probably because I have a constant UDP traffic rate which restart timeout counters
This is still happening using docker-ce-19.03.8-3.  
Note that on rhel 7.8 the command to see the conntrack entries is different.  
`cat /proc/net/nf_conntrack`

I had to yum install conntrack then issue 
`conntrack -D -p udp`

to get udp traffic to start flowing to the container again after a restart.
How about docker detect that it has launched a container that maps a udp port, and flushes the conntrack state for that port?
Could someone reopen this issue?
This issue still exists on engine version 20.10.9

Having low timeouts on conntrack table doesn't fix the problem for my setup:

net.netfilter.nf_conntrack_udp_timeout = 30
net.netfilter.nf_conntrack_udp_timeout_stream = 120

The only way is to flush the table with:

conntrack -D -p udp



Fix patch:
diff --git a/libnetwork/drivers/bridge/bridge.go b/libnetwork/drivers/bridge/bridge.go
index 0c7811c1ef143..0a886fe2b48ac 100644
--- a/libnetwork/drivers/bridge/bridge.go
+++ b/libnetwork/drivers/bridge/bridge.go
@@ -1352,6 +1352,11 @@ func (d *driver) ProgramExternalConnectivity(nid, eid string, options map[string
 		}
 	}()
 
+	// Clean the connection tracker state of the host for the
+	// specific endpoint. This is needed because some flows may be
+	// bound to the local proxy and won't bre redirect to the new endpoints.
+	clearEndpointConnections(d.nlh, endpoint)
+
 	if err = d.storeUpdate(endpoint); err != nil {
 		return fmt.Errorf("failed to update bridge endpoint %.7s to store: %v", endpoint.id, err)
 	}


Test patch: