Change:
We don't currently automatically restart etcd. We should.
Can we run etcd as a Pod without resource limit, so that we could benefit directly from 1) RestartPolicy: Always 2) resource monitoring and logging
When I bring up an e2e cluster (i.e. go run --up), without even running any of the e2e tests against it, it appears that monit is repeatedly killing and restarting etcd: davidopp more [UTC Feb 19 07:39:16] info : monit: generated unique Monit id and stored to '/var/lib/monit/id' [UTC Feb 19 07:39:16] info : Starting monit daemon [UTC Feb 19 07:39:16] error : State file '/var/lib/monit/state': Unable to read magic [UTC Feb 19 07:39:16] info : 'systeme2e-test-davidopp-master.c.striped-torus-760.internal' Monit started [UTC Feb 19 07:39:18] info : monit daemon with pid [5620] killed [UTC Feb 19 07:39:18] info : 'systeme2e-test-davidopp-master.c.striped-torus-760.internal' Monit stopped [UTC Feb 19 07:39:18] info : Starting monit daemon [UTC Feb 19 07:39:18] info : 'system_e2e-test-davidopp-master.c.striped-torus-760.internal' Monit started [UTC Feb 19 07:39:18] error : 'etcd' failed, cannot open a connection to INET[127.0.0.1:4001/v2/keys/] via TCP [UTC Feb 19 07:39:18] info : 'etcd' trying to restart [UTC Feb 19 07:39:18] info : 'etcd' stop: /etc/init.d/etcd [UTC Feb 19 07:39:18] info : 'etcd' start: /etc/init.d/etcd [UTC Feb 19 07:41:18] error : 'etcd' failed, cannot open a connection to INET[127.0.0.1:4001/v2/keys/] via TCP [UTC Feb 19 07:41:18] info : 'etcd' trying to restart [UTC Feb 19 07:41:18] info : 'etcd' stop: /etc/init.d/etcd [UTC Feb 19 07:41:18] info : 'etcd' start: /etc/init.d/etcd [UTC Feb 19 07:43:18] error : 'etcd' failed, cannot open a connection to INET[127.0.0.1:4001/v2/keys/] via TCP [UTC Feb 19 07:43:18] info : 'etcd' trying to restart [UTC Feb 19 07:43:18] info : 'etcd' stop: /etc/init.d/etcd [UTC Feb 19 07:43:18] info : 'etcd' start: /etc/init.d/etcd [UTC Feb 19 07:45:18] error : 'etcd' failed, cannot open a connection to INET[127.0.0.1:4001/v2/keys/] via TCP [UTC Feb 19 07:45:18] info : 'etcd' trying to restart [UTC Feb 19 07:45:18] info : 'etcd' stop: /etc/init.d/etcd [UTC Feb 19 07:45:18] info : 'etcd' start: /etc/init.d/etcd [...] I examined the etcd logs and there is nothing suspicious there. I suspect this may be the cause of some of the e2e test flakiness.
I think this particular problem is caused by firewall settings.
actually didn't fix the problem. Here's the [UTC Feb 19 23:22:25] info : monit: generated unique Monit id and stored to '/var/lib/monit/id' [UTC Feb 19 23:22:25] info : Starting monit daemon [UTC Feb 19 23:22:25] error : State file '/var/lib/monit/state': Unable to read magic [UTC Feb 19 23:22:25] info : 'systeme2e-test-davidopp-master.c.striped-torus-760.internal' Monit started [UTC Feb 19 23:22:26] info : monit daemon with pid [5481] killed [UTC Feb 19 23:22:26] info : 'systeme2e-test-davidopp-master.c.striped-torus-760.internal' Monit stopped [UTC Feb 19 23:22:27] info : Starting monit daemon [UTC Feb 19 23:22:27] info : 'system_e2e-test-davidopp-master.c.striped-torus-760.internal' Monit started [UTC Feb 19 23:22:32] error : 'etcd' failed, cannot open a connection to INET[kubernetes-master:4001/v2/keys/] via TCP [UTC Feb 19 23:22:32] info : 'etcd' trying to restart [UTC Feb 19 23:22:32] info : 'etcd' stop: /etc/init.d/etcd [UTC Feb 19 23:22:32] info : 'etcd' start: /etc/init.d/etcd [UTC Feb 19 23:24:37] error : 'etcd' failed, cannot open a connection to INET[kubernetes-master:4001/v2/keys/] via TCP [UTC Feb 19 23:24:37] info : 'etcd' trying to restart [UTC Feb 19 23:24:37] info : 'etcd' stop: /etc/init.d/etcd [UTC Feb 19 23:24:37] info : 'etcd' start: /etc/init.d/etcd [UTC Feb 19 23:26:43] error : 'etcd' failed, cannot open a connection to INET[kubernetes-master:4001/v2/keys/] via TCP [UTC Feb 19 23:26:43] info : 'etcd' trying to restart [UTC Feb 19 23:26:43] info : 'etcd' stop: /etc/init.d/etcd [UTC Feb 19 23:26:43] info : 'etcd' start: /etc/init.d/etcd [UTC Feb 19 23:28:48] error : 'etcd' failed, cannot open a connection to INET[kubernetes-master:4001/v2/keys/] via TCP [UTC Feb 19 23:28:48] info : 'etcd' trying to restart [UTC Feb 19 23:28:48] info : 'etcd' stop: /etc/init.d/etcd [UTC Feb 19 23:28:48] info : 'etcd' start: /etc/init.d/etcd [UTC Feb 19 23:30:53] error : 'etcd' failed, cannot open a connection to INET[kubernetes-master:4001/v2/keys/] via TCP [UTC Feb 19 23:30:53] info : 'etcd' trying to restart [UTC Feb 19 23:30:53] info : 'etcd' stop: /etc/init.d/etcd [UTC Feb 19 23:30:53] info : 'etcd' start: /etc/init.d/etcd
The reason the fix didn't work is that it hard-codes the hostname as kubernetes-master but the hostname may actually be anything. For example, in e2e test the master hostname is e2e-test-${USER}-master
This is fixed by changing etcd to startup with -bind-addr 0.0.0.0:4001 I'm sending a PR now.
(And thanks to for figuring out the problem was that etcd wasn't listening on the localhost address.)

Discussion:
<ins> check process etcd with pidfile /var/run/etcd.pid group etcd start program = "/etc/init.d/etcd start" stop program = "/etc/init.d/etcd stop" if failed host 127.0.0.1 port 4001 protocol http request "/v2/keys/" then restart </ins>

<ins> monit: pkg: - installed /etc/monit/conf.d/etcd: file: - managed - source: salt://monit/etcd - user: root - group: root - mode: 644 monit-service: service: - running - name: monit - watch: - pkg: monit - file: /etc/monit/conf.d/etcd </ins>

- kube-apiserver - kube-controller-manager - kube-scheduler <ins> - monit </ins> - nginx - kube-client-tools - logrotate