Issue 1:
Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/.): No

What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.):

"crypto/tls: failed to parse certificate PEM data"

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT

Kubernetes version (use kubectl version): v1.2.4

Server Version: version.Info{Major:"1", Minor:"2", GitVersion:"v1.2.4+coreos.cni.1", GitCommit:"89e3af04ef436f049e2e2d169e57ad1712cac375", GitTreeState:"clean"

Environment:

Cloud provider or hardware configuration: AWS EC2 Instance

OS (e.g. from /etc/os-release): CoreOS 1010.4.0 (MoreOS)

Kernel (e.g. uname -a): Linux ip-10-0-2-42.ec2.internal 4.5.0-coreos-r1 #2 SMP Tue May 24 17:45:36 UTC 2016 x86_64 Intel(R) Xeon(R) CPU E5-2686 v4 @ 2.30GHz GenuineIntel GNU/Linux

Install tools: kube-aws

Others:

What happened:

The kubelet service failed (reason unknown)

Attempting to restart kubelet failed, with:

crypto/tls: failed to parse certificate PEM data

See full logs and rambling thoughts below.

What you expected to happen:

Expected kubelet not to fail, and for kubelet to restart, if it did fail.

How to reproduce it (as minimally and precisely as possible):

?

Anything else do we need to know:

Full logs and rambling thoughts:

After noticing that the node was NotReady

$ journalctl -u kubelet
-- Logs begin at Sun 2016-10-16 07:16:48 UTC, end at Mon 2016-10-17 14:01:01 UTC. --
Oct 16 12:01:27 ip-10-0-2-42.ec2.internal kubelet-wrapper[1276]: E1016 12:01:27.793814 1276 fsHandler.go:106] failed to collect filesystem stats - du command failed on /var/lib/docker/overlay/d5db24c3fbe5fbfbdbeaab9d479958c352a0a8740298
Oct 16 12:01:27 ip-10-0-2-42.ec2.internal kubelet-wrapper[1276]: , stderr: du: cannot access '/var/lib/docker/overlay/d5db24c3fbe5fbfbdbeaab9d479958c352a0a874029848cc31e0a97d686a9bd3/upper/tmp/tmp-5039jZ3BGus1nfDW': No such file or directory
Oct 16 12:01:27 ip-10-0-2-42.ec2.internal kubelet-wrapper[1276]: - exit status 1
Oct 16 12:48:40 ip-10-0-2-42.ec2.internal kubelet-wrapper[1276]: E1016 12:48:40.543457 1276 event.go:193] Server rejected event '&api.Event{TypeMeta:unversioned.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:api.ObjectMeta{Name:"kube-dns-
Oct 16 18:55:04 ip-10-0-2-42.ec2.internal kubelet-wrapper[1276]: W1016 18:55:04.175757 1276 request.go:627] Throttling request took 148.06449ms, request: GET:https://10.0.4.50:443/api/v1/nodes/ip-10-0-2-42.ec2.internal
Oct 16 18:58:15 ip-10-0-2-42.ec2.internal systemd[1]: kubelet.service: Service hold-off time over, scheduling restart.
Oct 16 18:58:15 ip-10-0-2-42.ec2.internal systemd[1]: Stopped kubelet.service.
Oct 16 18:58:18 ip-10-0-2-42.ec2.internal systemd[1]: Dependency failed for kubelet.service.
Oct 16 18:58:18 ip-10-0-2-42.ec2.internal systemd[1]: kubelet.service: Job kubelet.service/start failed with result 'dependency'.

I'm not sure what caused kubelet to shut down. I see the failed to collect filesystem stats error, but it looks like that may be a red herring, possibly caused by a custom docker garbage collection script we are running (because the built in cleanup services are not working for us).

I see the Job kubelet.service/start failed with result 'dependency'. So check the docker service

$ journalctl -u docker
-- Logs begin at Sun 2016-10-16 07:16:48 UTC, end at Mon 2016-10-17 14:01:41 UTC. --
Oct 16 18:58:05 ip-10-0-2-42.ec2.internal systemd[1]: Stopping Docker Application Container Engine...
Oct 16 18:58:15 ip-10-0-2-42.ec2.internal systemd[1]: Stopped Docker Application Container Engine.
Oct 16 18:58:18 ip-10-0-2-42.ec2.internal systemd[1]: Dependency failed for Docker Application Container Engine.
Oct 16 18:58:18 ip-10-0-2-42.ec2.internal systemd[1]: docker.service: Job docker.service/start failed with result 'dependency'.
Oct 17 00:00:14 ip-10-0-2-42.ec2.internal systemd[1]: Started Docker Application Container Engine.
Oct 17 00:00:26 ip-10-0-2-42.ec2.internal dockerd[28294]: time="2016-10-17T00:00:26.249045066Z" level=info msg="Graph migration to content-addressability took 0.00 seconds"
Oct 17 00:00:26 ip-10-0-2-42.ec2.internal dockerd[28294]: time="2016-10-17T00:00:26.252948977Z" level=info msg="Firewalld running: false"
Oct 17 00:00:26 ip-10-0-2-42.ec2.internal dockerd[28294]: time="2016-10-17T00:00:26.566883833Z" level=info msg="Loading containers: start."
Oct 17 00:00:26 ip-10-0-2-42.ec2.internal dockerd[28294]: .......................................................time="2016-10-17T00:00:26.766104356Z" level=error msg="Error unmounting container 9c20ac08b141a474b8ba76698b339729907d35a6a286
Oct 17 00:00:26 ip-10-0-2-42.ec2.internal dockerd[28294]: time="2016-10-17T00:00:26.767774056Z" level=error msg="Error unmounting container e3274fd20b73356d2b9d4cf94cb2e5028b0cfdf46151002fd71af5bd86e4b14e: not mounted"
Oct 17 00:00:26 ip-10-0-2-42.ec2.internal dockerd[28294]: time="2016-10-17T00:00:26.771743362Z" level=error msg="Error unmounting container ea6d0e45e10db34b7b934a9307f1791336b258f125dee256d0aa9611363c5261: not mounted"
Oct 17 00:00:26 ip-10-0-2-42.ec2.internal dockerd[28294]: time="2016-10-17T00:00:26.773098949Z" level=error msg="Error unmounting container 7ad2c477853f1429ebefd7687af2b60ac73a83b2590fd52685e139dc073e1f6e: not mounted"
Oct 17 00:00:26 ip-10-0-2-42.ec2.internal dockerd[28294]: time="2016-10-17T00:00:26.777013970Z" level=error msg="Error unmounting container f2448c82bc67eb7b932b3ace75a878272ef917a09e37e542e7a540821815357b: not mounted"
Oct 17 00:00:26 ip-10-0-2-42.ec2.internal dockerd[28294]: time="2016-10-17T00:00:26.788810168Z" level=error msg="Error unmounting container 2ecdff73aedd2c17b57f784fc812ed715240627e37ff9b2ca6b6eb0fa323e757: not mounted"
Oct 17 00:00:26 ip-10-0-2-42.ec2.internal dockerd[28294]: time="2016-10-17T00:00:26.799454818Z" level=error msg="Error unmounting container f0e337d6fd15884d6dd8c69d7b8941232a332054a58d2c66ebc99eac0190a477: not mounted"
Oct 17 00:00:26 ip-10-0-2-42.ec2.internal dockerd[28294]: time="2016-10-17T00:00:26.801740726Z" level=error msg="Error unmounting container 512a8344fee1bb62bcddfa3327a03ea886ccf46fdd9c46f1a0d872ff386004b9: not mounted"
Oct 17 00:00:26 ip-10-0-2-42.ec2.internal dockerd[28294]: time="2016-10-17T00:00:26.804142833Z" level=error msg="Error unmounting container 5818f58156e378425cb2aa48e9646d0d35b86e89338616366b8b21f2fbe58e25: not mounted"
Oct 17 00:00:26 ip-10-0-2-42.ec2.internal dockerd[28294]: time="2016-10-17T00:00:26.805297087Z" level=error msg="Error unmounting container 20a4cda200c33d6263a540caba028a373ecdf5ca2733e40e32627efd71f38d79: not mounted"
Oct 17 00:00:26 ip-10-0-2-42.ec2.internal dockerd[28294]: time="2016-10-17T00:00:26.807651272Z" level=error msg="Error unmounting container 5e11bedd3511b33a1b16ec42af039c0f7a4959c86092f543f2f83816b564ce3f: not mounted"
Oct 17 00:00:26 ip-10-0-2-42.ec2.internal dockerd[28294]: time="2016-10-17T00:00:26.808670518Z" level=error msg="Error unmounting container 63d415b23d7d8d4f399f32d8e7cfa27148af80aa6ba3c1164cfbb43ce94f6a7e: not mounted"
Oct 17 00:00:26 ip-10-0-2-42.ec2.internal dockerd[28294]: time="2016-10-17T00:00:26.809797868Z" level=error msg="Error unmounting container f2199232be2d0f6f93f39e311970222e2d2e18bd8be62c4561304f9dcc58af3d: not mounted"
Oct 17 00:00:26 ip-10-0-2-42.ec2.internal dockerd[28294]: time="2016-10-17T00:00:26.812997843Z" level=error msg="Error unmounting container e260b44e03890339a4de616259d42b6f4529fa0a19e8e259b781d4abf7e68d1e: not mounted"
Oct 17 00:00:26 ip-10-0-2-42.ec2.internal dockerd[28294]: time="2016-10-17T00:00:26.816434872Z" level=error msg="Error unmounting container c23aa4d3db9cba77ee37d5debed98e66a09039ecb78c191c4372c187d9e69c3c: not mounted"
Oct 17 00:00:26 ip-10-0-2-42.ec2.internal dockerd[28294]: time="2016-10-17T00:00:26.817642230Z" level=error msg="Error unmounting container 0c0a0d62ca3af74660f6d671510823485cc1a37bb19e8718e93b28e4cf0c7e4f: not mounted"
Oct 17 00:00:26 ip-10-0-2-42.ec2.internal dockerd[28294]: time="2016-10-17T00:00:26.818783758Z" level=error msg="Error unmounting container 3015b62493fa80462a97336503fb7749cb5a28b37cf70073812b3d37fb806128: not mounted"
Oct 17 00:00:26 ip-10-0-2-42.ec2.internal dockerd[28294]: time="2016-10-17T00:00:26.825780463Z" level=error msg="Error unmounting container 8b549e1ffcd94240d9d7469bc20c400160b2b7ff9a8a3c44eccf48de8f62c141: not mounted"
Oct 17 00:00:26 ip-10-0-2-42.ec2.internal dockerd[28294]: time="2016-10-17T00:00:26.868323368Z" level=error msg="Error unmounting container caf802a7a506053a1f5eef98ca37a017f0203dffcc0eae68019c5288e165ac61: not mounted"
Oct 17 00:00:26 ip-10-0-2-42.ec2.internal dockerd[28294]: time="2016-10-17T00:00:26.873743087Z" level=error msg="Error unmounting container 66dbce340f4e2e0541c814ce224ed059fa99f13cd274a94b6e4ce6ab3a148e1d: not mounted"
Oct 17 00:00:26 ip-10-0-2-42.ec2.internal dockerd[28294]: time="2016-10-17T00:00:26.884863245Z" level=info msg="Loading containers: done."
Oct 17 00:00:26 ip-10-0-2-42.ec2.internal dockerd[28294]: time="2016-10-17T00:00:26.884879420Z" level=info msg="Daemon has completed initialization"
Oct 17 00:00:26 ip-10-0-2-42.ec2.internal dockerd[28294]: time="2016-10-17T00:00:26.884896942Z" level=info msg="Docker daemon" commit=8acee1b execdriver=native-0.2 graphdriver=overlay version=1.10.3
Oct 17 00:00:26 ip-10-0-2-42.ec2.internal dockerd[28294]: time="2016-10-17T00:00:26.890880322Z" level=info msg="API listen on /var/run/docker.sock"

I see a docker.service: Job docker.service/start failed with result 'dependency'. log in there. Maybe it's flannel?

$ journalctl -u flanneld
-- Logs begin at Sun 2016-10-16 07:16:48 UTC, end at Mon 2016-10-17 14:53:22 UTC. --
Oct 16 07:16:48 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:48.080742 00001 device.go:187] calling NeighSet: 10.2.58.14, 96:75:16:ae:1e:ce
Oct 16 07:16:48 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:48.080863 00001 vxlan.go:356] AddL3 succeeded
Oct 16 07:16:48 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:48.407509 00001 vxlan.go:340] Ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.0
Oct 16 07:16:48 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:48.832713 00001 vxlan.go:340] Ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.10
Oct 16 07:16:49 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:49.408852 00001 vxlan.go:340] Ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.6
Oct 16 07:16:49 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:49.856828 00001 vxlan.go:340] Ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.9
Oct 16 07:16:50 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:50.032862 00001 vxlan.go:340] Ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.3
Oct 16 07:16:50 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:50.410953 00001 vxlan.go:340] Ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.0
Oct 16 07:16:50 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:50.816900 00001 vxlan.go:340] Ignoring not a miss: be:66:1c:f3:6f:bf, 10.2.59.2
Oct 16 07:16:50 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:50.896856 00001 vxlan.go:340] Ignoring not a miss: be:66:1c:f3:6f:bf, 10.2.59.8
Oct 16 07:16:52 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:52.752864 00001 vxlan.go:340] Ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.2
Oct 16 07:16:53 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:53.035102 00001 vxlan.go:340] Ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.0
Oct 16 07:16:53 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:53.038654 00001 vxlan.go:340] Ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.3
Oct 16 07:16:53 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:53.633088 00001 vxlan.go:340] Ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.4
Oct 16 07:16:53 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:53.902816 00001 vxlan.go:340] Ignoring not a miss: be:66:1c:f3:6f:bf, 10.2.59.8
Oct 16 07:16:54 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:54.040702 00001 vxlan.go:340] Ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.3
Oct 16 07:16:54 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:54.080715 00001 vxlan.go:340] Ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.10
Oct 16 07:16:54 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:54.720732 00001 vxlan.go:340] Ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.6
Oct 16 07:16:54 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:54.896893 00001 vxlan.go:340] Ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.9
Oct 16 07:16:54 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:54.904658 00001 vxlan.go:340] Ignoring not a miss: be:66:1c:f3:6f:bf, 10.2.59.8
Oct 16 07:16:55 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:55.042682 00001 vxlan.go:340] Ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.3
Oct 16 07:16:55 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:55.427159 00001 vxlan.go:345] L3 miss: 10.2.59.9
Oct 16 07:16:55 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:55.427208 00001 device.go:187] calling NeighSet: 10.2.59.9, be:66:1c:f3:6f:bf
Oct 16 07:16:55 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:55.427391 00001 vxlan.go:356] AddL3 succeeded
Oct 16 07:16:55 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:55.758828 00001 vxlan.go:340] Ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.2
Oct 16 07:16:55 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:55.906672 00001 vxlan.go:340] Ignoring not a miss: be:66:1c:f3:6f:bf, 10.2.59.8
Oct 16 07:16:56 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:56.395214 00001 vxlan.go:345] L3 miss: 10.2.58.3
Oct 16 07:16:56 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:56.395269 00001 device.go:187] calling NeighSet: 10.2.58.3, 96:75:16:ae:1e:ce
Oct 16 07:16:56 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:56.395376 00001 vxlan.go:356] AddL3 succeeded
Oct 16 07:16:56 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:56.560665 00001 vxlan.go:340] Ignoring not a miss: be:66:1c:f3:6f:bf, 10.2.59.2
Oct 16 07:16:56 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:56.760705 00001 vxlan.go:340] Ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.2
Oct 16 07:16:56 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:56.996054 00001 vxlan.go:345] L3 miss: 10.2.59.8
Oct 16 07:16:56 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:56.996108 00001 device.go:187] calling NeighSet: 10.2.59.8, be:66:1c:f3:6f:bf
Oct 16 07:16:56 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:56.996212 00001 vxlan.go:356] AddL3 succeeded
Oct 16 07:16:57 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:57.086698 00001 vxlan.go:340] Ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.10
Oct 16 07:16:57 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:57.726688 00001 vxlan.go:340] Ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.6
Oct 16 07:16:57 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:57.762682 00001 vxlan.go:340] Ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.2
Oct 16 07:16:57 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:57.902809 00001 vxlan.go:340] Ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.9
Oct 16 07:16:58 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:58.088875 00001 vxlan.go:340] Ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.10
Oct 16 07:16:58 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:58.727838 00001 vxlan.go:340] Ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.0
Oct 16 07:16:58 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:58.729197 00001 vxlan.go:340] Ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.6
Oct 16 07:16:58 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:58.752964 00001 vxlan.go:340] Ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.4
Oct 16 07:16:58 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:58.904853 00001 vxlan.go:340] Ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.9
Oct 16 07:16:59 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:59.090678 00001 vxlan.go:340] Ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.10
Oct 16 07:16:59 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:59.566701 00001 vxlan.go:340] Ignoring not a miss: be:66:1c:f3:6f:bf, 10.2.59.2
Oct 16 07:16:59 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:59.728951 00001 vxlan.go:340] Ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.0
Oct 16 07:16:59 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:59.730772 00001 vxlan.go:340] Ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.6
Oct 16 07:16:59 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:16:59.906840 00001 vxlan.go:340] Ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.9
Oct 16 07:17:00 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:17:00.112784 00001 vxlan.go:345] L3 miss: 10.2.58.10
Oct 16 07:17:00 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:17:00.112832 00001 device.go:187] calling NeighSet: 10.2.58.10, 96:75:16:ae:1e:ce
Oct 16 07:17:00 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:17:00.112943 00001 vxlan.go:356] AddL3 succeeded
Oct 16 07:17:00 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:17:00.568835 00001 vxlan.go:340] Ignoring not a miss: be:66:1c:f3:6f:bf, 10.2.59.2
Oct 16 07:17:00 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:17:00.731082 00001 vxlan.go:340] Ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.0
Oct 16 07:17:01 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:17:01.570500 00001 vxlan.go:340] Ignoring not a miss: be:66:1c:f3:6f:bf, 10.2.59.2
Oct 16 07:17:01 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:17:01.758877 00001 vxlan.go:340] Ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.4
Oct 16 07:17:02 ip-10-0-2-42.ec2.internal sdnotify-proxy[901]: I1016 07:17:02.027092 00001 vxlan.go:340] Ignoring not a miss: 96:75:16:ae:1e:ce, 10.2.58.0

Don't see any fatal errors there...

So I try restarting kubelet:

$ servicectl restart kubelet
$ journalctl -fu kubelet
Oct 17 14:02:37 ip-10-0-2-42.ec2.internal kubelet-wrapper[7992]: I1017 14:02:37.887527 7992 manager.go:123] Starting to sync pod status with apiserver
Oct 17 14:02:37 ip-10-0-2-42.ec2.internal kubelet-wrapper[7992]: I1017 14:02:37.887555 7992 kubelet.go:2356] Starting kubelet main sync loop.
Oct 17 14:02:37 ip-10-0-2-42.ec2.internal kubelet-wrapper[7992]: I1017 14:02:37.887568 7992 kubelet.go:2365] skipping pod synchronization - [container runtime is down]
Oct 17 14:02:37 ip-10-0-2-42.ec2.internal kubelet-wrapper[7992]: F1017 14:02:37.888926 7992 server.go:118] crypto/tls: failed to parse certificate PEM data
Oct 17 14:02:37 ip-10-0-2-42.ec2.internal systemd[1]: kubelet.service: Main process exited, code=exited, status=255/n/a
Oct 17 14:02:37 ip-10-0-2-42.ec2.internal systemd[1]: kubelet.service: Unit entered failed state.
Oct 17 14:02:37 ip-10-0-2-42.ec2.internal systemd[1]: kubelet.service: Failed with result 'exit-code'.

That crypto/tls: failed to parse certificate PEM data message looks ominous.

I heard from a colleague that on a different node, the RSA keys just disappeared. He was able to fix the issue by copying them in from another server. I don't know if that's what's going on here, but just a guess... (and he's out of town, so I can't verify)

All of our pods were quickly relocated to a functioning node, so the impact wasn't too heavy. But it would be nice to know what's going on, and how I could fix/prevent this in the future.

Issue 2:
As discussed in https://groups.google.com/forum/#!topic/google-containers/vA8vIfZ7WUA, we can make some small changes to allow one yaml file contain multiple resources to reduce the file amount of a complex system.