Issue 1:
We've been encountering increasing number of issues where the kubelet seems to be in some sort of hung state while handling processing events (like creating or destroying a pod):

$ kubectl get po
NAME READY STATUS
appname-2331821309-lgj2o 0/1 ContainerCreating

Getting on the kubelet where the container is hung, the logs seem to have a lot of transport errors (maybe our kube-apiserver crashed or disappeared):

kubelet.log:

kubelet: E0613 11:01:53.943206 28599 reflector.go:271] pkg/kubelet/config/apiserver.go:43: Failed to watch *api.Pod: Get https://kube-apiserver:6443/api/v1/watch/pods?fieldSelector=spec.nodeName%3Dkubelet01&resourceVersion=6439574&timeoutSeconds=420: dial tcp 10.0.0.1:6443: connection refused
kubelet: E0613 11:01:53.943276 28599 reflector.go:271] pkg/kubelet/kubelet.go:240: Failed to watch *api.Service: Get https://kube-apiserver:6443/api/v1/watch/services?resourceVersion=6439299&timeoutSeconds=487: dial tcp 10.0.0.1:6443: connection refused
kubelet: E0613 11:01:53.943315 28599 reflector.go:271] pkg/kubelet/kubelet.go:259: Failed to watch *api.Node: Get https://kube-apiserver:6443/api/v1/watch/nodes?fieldSelector=metadata.name%3Dkubelet01&resourceVersion=6468007&timeoutSeconds=505: dial tcp 10.0.0.1:6443: connection refused
kubelet: E0613 11:01:55.640953 28599 reflector.go:271] pkg/kubelet/config/apiserver.go:43: Failed to watch *api.Pod: too old resource version: 6439574 (6468022)
kubelet: E0613 11:01:55.667667 28599 reflector.go:271] pkg/kubelet/kubelet.go:240: Failed to watch *api.Service: too old resource version: 6439299 (6468022)
kubelet: E0613 11:01:55.843807 28599 reflector.go:271] pkg/kubelet/kubelet.go:259: Failed to watch *api.Node: too old resource version: 6468007 (6468022)
kubelet: E0613 11:09:54.814446 28599 reflector.go:271] pkg/kubelet/config/apiserver.go:43: Failed to watch *api.Pod: too old resource version: 6468023 (6468279)
kubelet: E0613 15:34:56.610783 28599 reflector.go:271] pkg/kubelet/kubelet.go:259: Failed to watch *api.Node: Get https://kube-apiserver:6443/api/v1/watch/nodes?fieldSelector=metadata.name%3Dkubelet01&resourceVersion=6496591&timeoutSeconds=398: dial tcp 10.0.0.1:6443: connection refused
kubelet: E0613 15:34:56.615108 28599 reflector.go:271] pkg/kubelet/config/apiserver.go:43: Failed to watch *api.Pod: Get https://kube-apiserver:6443/api/v1/watch/pods?fieldSelector=spec.nodeName%3Dkubelet01&resourceVersion=6468845&timeoutSeconds=374: dial tcp 10.0.0.1:6443: connection refused
kubelet: E0613 15:34:56.615166 28599 reflector.go:271] pkg/kubelet/kubelet.go:240: Failed to watch *api.Service: Get https://kube-apiserver:6443/api/v1/watch/services?resourceVersion=6468023&timeoutSeconds=532: dial tcp 10.0.0.1:6443: connection refused
kubelet: E0613 15:34:57.613588 28599 reflector.go:271] pkg/kubelet/kubelet.go:259: Failed to watch *api.Node: Get https://kube-apiserver:6443/api/v1/watch/nodes?fieldSelector=metadata.name%3Dkubelet01&resourceVersion=6496591&timeoutSeconds=405: dial tcp 10.0.0.1:6443: connection refused
kubelet: E0613 15:34:57.617199 28599 reflector.go:271] pkg/kubelet/kubelet.go:240: Failed to watch *api.Service: Get https://kube-apiserver:6443/api/v1/watch/services?resourceVersion=6468023&timeoutSeconds=429: dial tcp 10.0.0.1:6443: connection refused
kubelet: E0613 15:34:57.617255 28599 reflector.go:271] pkg/kubelet/config/apiserver.go:43: Failed to watch *api.Pod: Get https://kube-apiserver:6443/api/v1/watch/pods?fieldSelector=spec.nodeName%3Dkubelet01&resourceVersion=6468845&timeoutSeconds=526: dial tcp 10.0.0.1:6443: connection refused
kubelet: E0613 15:34:58.786094 28599 reflector.go:271] pkg/kubelet/kubelet.go:259: Failed to watch *api.Node: too old resource version: 6496591 (6496612)
kubelet: E0613 15:34:58.969776 28599 reflector.go:271] pkg/kubelet/kubelet.go:240: Failed to watch *api.Service: too old resource version: 6468023 (6496611)
kubelet: E0613 15:34:59.037563 28599 reflector.go:271] pkg/kubelet/config/apiserver.go:43: Failed to watch *api.Pod: too old resource version: 6468845 (6496611)
kubelet: E0613 15:35:00.512631 28599 reflector.go:271] pkg/kubelet/config/apiserver.go:43: Failed to watch *api.Pod: too old resource version: 6496612 (6496617)

Restarting the kubelet process seems to make everything process as expected.

While it's definitely not ideal (the apiserver being unavailable) I would not expect what is a very possible intermittent network failure to result in a completely unresponsive kubelet. Instead I'd expect it to gracefully resolve itself and resume communication with the apiserver or to crash altogether. The latter would be a lot easier from an operators standpoint as at least we would have some idea there is an issue occurring.

Sorry in advance for the sparse information in this issue. I'm happy to provide as much as possible but the issue has been pretty intermittent and the kubelet logs don't seem overly helpful.

Issue 2:
Following this doc https://github.com/kubernetes/kubernetes/blob/master/cluster/libvirt-coreos/README.md

git clone https://github.com/kubernetes/kubernetes/
cd kubernetes
make release-skip-tests
export KUBERNETES_PROVIDER=libvirt-coreos
./cluster/kube-up.sh
ssh core@192.168.10.2
docker run -d --name nginx nginx
docker inspect --format '{{ .NetworkSettings.IPAddress }}' nginx # i.e. returns 10.10.1.3

then run curl on another k8s minion:

ssh core@192.168.10.3
curl 10.10.1.3

And you'll see result below:

core@kubernetes-minion-2 ~ $ curl 10.10.1.3
curl: (56) Recv failure: Connection reset by peer
core@kubernetes-minion-2 ~ $ curl 10.10.1.3
curl: (56) Recv failure: Connection reset by peer
core@kubernetes-minion-2 ~ $ curl 10.10.1.3
curl: (56) Recv failure: Connection reset by peer
core@kubernetes-minion-2 ~ $ curl 10.10.1.3
curl: (56) Recv failure: Connection reset by peer
core@kubernetes-minion-2 ~ $ curl 10.10.1.3
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
body {
width: 35em;
margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif;
}
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>
core@kubernetes-minion-2 ~ $ curl 10.10.1.3
curl: (56) Recv failure: Connection reset by peer

So looks like there is something wrong with the bridge configuration in libvirt-coreos provider.

Here is tcpdump:

10:50:44.059742 IP 10.10.2.1.59138 > 10.10.1.3.http: Flags [S], seq 3020067180, win 29200, options [mss 1460,sackOK,TS val 3556468 ecr 0,nop,wscale 6], length 0
10:50:44.059777 IP 10.10.1.3.http > 10.10.2.1.59138: Flags [S.], seq 1880978123, ack 3020067181, win 28960, options [mss 1460,sackOK,TS val 3560072 ecr 3556468,nop,wscale 6], length 0
10:50:44.059793 IP 10.10.1.3.http > 10.10.2.1.59138: Flags [S.], seq 1880978123, ack 3020067181, win 28960, options [mss 1460,sackOK,TS val 3560072 ecr 3556468,nop,wscale 6], length 0
10:50:44.060083 IP 10.10.2.1.59138 > 10.10.1.3.http: Flags [.], ack 1, win 457, options [nop,nop,TS val 3556468 ecr 3560072], length 0
10:50:44.060090 IP 10.10.2.1.59138 > 10.10.1.3.http: Flags [P.], seq 1:74, ack 1, win 457, options [nop,nop,TS val 3556468 ecr 3560072], length 73
10:50:44.060110 IP 10.10.1.3.http > 10.10.2.1.59138: Flags [R], seq 1880978124, win 0, length 0
10:50:44.060114 IP 10.10.1.3.http > 10.10.2.1.59138: Flags [R], seq 1880978124, win 0, length 0
10:50:44.060131 IP 10.10.1.3.http > 10.10.2.1.59138: Flags [R], seq 1880978124, win 0, length 0
10:50:44.060133 IP 10.10.1.3.http > 10.10.2.1.59138: Flags [R], seq 1880978124, win 0, length 0

/cc @lhuard1A