Issue 1:
I created an e2e cluster with REGISTER_MASTER_KUBELET=true, and the master node is extremely unstable with docker/kubelet being restarted frequently.

I ended up fixing this by removing /etc/supervisor/conf.d/docker.conf to prevent docker from being restarted again. The cluster was stable and functioning after the fix.

Here are my questions:

Is REGISTER_MASTER_KUBELET=true supposed to work in this case, or was there any other configurations that I needed to change?

Do we run any e2e suite with master node registered?

The docker health check performed by supervisord seems defective on the master node in this case, but I am not sure why.

/cc @roberthbailey

Issue 2:
Running kube-up on a clean build of 6afbaf6, I'm seeing the docker on the master crashing, and kube-apiserver crash with it.

This is the standard 4-node GCE cluster, I haven't touched any config.

kubernetes·(6afbaf6...) $ make clean release-skip-tests && kup && kdown
[...]
Waiting for cluster initialization.

This will continually check to see if the API for kubernetes is reachable.
This might loop forever if there was some uncaught error during start
up.

................................Kubernetes cluster created.
Wrote config for devbox-964_kubernetes to $HOME/.kube/config

Kubernetes cluster is running. The master is running at:

https://IP

The user name and password to use is located in $HOME/.kube/config.

... calling validate-cluster
Waiting for 5 ready nodes. 1 ready nodes, 5 registered. Retrying.
Waiting for 5 ready nodes. 1 ready nodes, 5 registered. Retrying.
Waiting for 5 ready nodes. 1 ready nodes, 5 registered. Retrying.
error: couldn't read version from server: Get https://IP/api: dial tcp IP:443: connection refused

List of all pods:

kubernetes·(6afbaf6...) $ k get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system etcd-server-kubernetes-master 1/1 Running 3 16m
kube-system fluentd-cloud-logging-kubernetes-master 1/1 Running 3 16m
kube-system fluentd-cloud-logging-kubernetes-minion-5h5s 1/1 Running 0 15m
kube-system fluentd-cloud-logging-kubernetes-minion-e3p1 1/1 Running 0 15m
kube-system fluentd-cloud-logging-kubernetes-minion-k0jp 1/1 Running 0 15m
kube-system fluentd-cloud-logging-kubernetes-minion-ykur 1/1 Running 0 15m
kube-system kube-apiserver-kubernetes-master 1/1 Running 3 16m
kube-system kube-controller-manager-kubernetes-master 1/1 Running 3 16m
kube-system kube-dns-v8-v7svt 4/4 Running 0 16m
kube-system kube-scheduler-kubernetes-master 1/1 Running 3 16m
kube-system kube-ui-v1-65m9j 0/1 Image: gcr.io/google_containers/kube-ui:v1.1 is not ready on the node 0 16m
kube-system monitoring-heapster-v7-3xn8i 0/1 InsufficientFreeCPU 0 16m
kube-system monitoring-heapster-v7-6h6d2 0/1 InsufficientFreeCPU 0 16m
kube-system monitoring-heapster-v7-79ozz 0/1 InsufficientFreeCPU 0 16m
kube-system monitoring-heapster-v7-8jffe 0/1 InsufficientFreeCPU 0 16m
kube-system monitoring-heapster-v7-95i3o 0/1 InsufficientFreeCPU 0 16m
kube-system monitoring-heapster-v7-aur4k 0/1 Running 9 15m
kube-system monitoring-heapster-v7-e7wwf 0/1 InsufficientFreeCPU 0 16m
kube-system monitoring-heapster-v7-g1px0 0/1 InsufficientFreeCPU 0 16m
kube-system monitoring-heapster-v7-gbyfh 0/1 InsufficientFreeCPU 0 16m
kube-system monitoring-heapster-v7-mppao 0/1 InsufficientFreeCPU 0 16m
kube-system monitoring-influx-grafana-v1-421pp 0/2 InsufficientFreeCPU 0 16m
kube-system monitoring-influx-grafana-v1-qhvpi 2/2 Running 0 15m

Not sure why so many heapster pods are scheduled.

Watching the status of the apiserver and docker on the master:

uluyol@kubernetes-master:~$ while true; do pgrep kube-apiserver &>/dev/null; kube_up=$?; [[ $(pgrep docker | wc -l) == 2 ]]; docker_up=$?; if [[ $kube_up == 0 ]]; then echo -n apiserver up; else echo -n apiserver down; fi; if [[ $docker_up == 0 ]]; then echo ' docker up'; else echo ' docker down'; fi; sleep 1; done
apiserver down docker down
apiserver down docker down
apiserver down docker down
apiserver down docker down
apiserver down docker down
apiserver down docker down
apiserver down docker down
apiserver down docker up
apiserver down docker down
apiserver up docker up
apiserver up docker up
[ +35 more ]
apiserver up docker up
apiserver down docker down
apiserver down docker down
apiserver down docker down
[...]

After digging around with @mikedanese on Friday, looks like removing the kubelet_api_servers definition in /etc/salt/minion.d/grains.conf fixes this (I followed this by calling salt-call state.highstate). I have 25 min of apiserver/docker uptime so far, and the scheduled pods looks more like what it's supposed to

kubernetes·(6afbaf6...) $ k get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system fluentd-cloud-logging-kubernetes-minion-5h5s 1/1 Running 0 51m
kube-system fluentd-cloud-logging-kubernetes-minion-e3p1 1/1 Running 0 51m
kube-system fluentd-cloud-logging-kubernetes-minion-k0jp 1/1 Running 0 51m
kube-system fluentd-cloud-logging-kubernetes-minion-ykur 1/1 Running 0 51m
kube-system kube-dns-v8-v7svt 4/4 Running 0 52m
kube-system kube-ui-v1-kwrq6 1/1 Running 0 25m
kube-system monitoring-heapster-v7-aur4k 0/1 Running 10 51m
kube-system monitoring-influx-grafana-v1-qhvpi 2/2 Running 0 51m

It seems the 8df33bc was the commit that changed this. EDIT: For clarification, I mean that this commit is when we started setting kubelet_api_servers on the master.