Change:
Node 2's kubelet logs: controller-manager side: I have a cluster to hand off if someone wants to debug. It's not a transient state: node 2 just failed to ever post a valid status. cc cc (who looked at it briefly with me)
I will take a look soon.
Ok, why kubelet on node 2 talks to ip address of master instead of hostname? root ps -ef | grep kubelet root 1 3 17:35 ? 00:10:45 /usr/local/bin/kubelet --apiservers=https://10.240.49.47 --config=/etc/kubernetes/manifests --allowprivileged=False --v=2 --clusterdns=10.51.240.10 -- It looks like a cluster up issue to me, not a node problem.
It looks like failed to translate the master name into the fqdn (which is done ).
Fascinating. I think we're seeing instance-DNS flakiness, because that name is supposed to get converted ASAP. But what's weirder is ... that's not even the master:
The is definitely right:
Okay, this is messed up. Here's node 1, which is happy: Here's our unhappy node .... that decided to bring itself as a master:
haha... I checked kube-env and did suspect unhappy node treat itself as master node based on those variables. But I didn't check salt state :-)
I ... just can't get there. By the time we get here: , node-2 has set: My best guess is that flaked here: . I think we're seeing a systemic issue where we're racing with GCE network configuration.
I'm going to work up a PR that hopefully improves the reliability of this section to some extent.
<a href=""<img src="" title="made at "/</a
My working theory is that we raced with the configuration done normally by the GCE NetworkManager, leaving us in a bogus state when we made that check.
Why are we relying on inferring the master/node status from the hostname. It should be obvious from the variables set in the kube-env if you are the master or not. Masters don't get passed so why not just check that? Making sure DNS is working is probably also good for reliability...
On GKE, we actually pass the same kube-env on both sides for simplicity. We can obviously change that, but I was going for a change that didn't involve a deployment change on our side, too.

Discussion:
readonly KNOWN_TOKENS_FILE="/srv/salt-overlay/salt/kube-apiserver/known_tokens.csv" readonly BASIC_AUTH_FILE="/srv/salt-overlay/salt/kube-apiserver/basic_auth.csv" <ins> function ensure-basic-networking() { # Deal with GCE networking bring-up race. (We rely on DNS for a lot, # and it's just not worth doing a whole lot of startup work if this # isn't ready yet.) until getent hosts metadata.google.internal &>/dev/null; do echo 'Waiting for functional DNS (trying to resolve metadata.google.internal)...' sleep 3 done until getent hosts $(hostname -f) &>/dev/null; do echo 'Waiting for functional DNS (trying to resolve my own FQDN)...' sleep 3 done until getent hosts $(hostname -i) &>/dev/null; do echo 'Waiting for functional DNS (trying to resolve my own IP)...' sleep 3 done echo "Networking functional on $(hostname) ($(hostname -i))" } </ins> function ensure-install-dir() { INSTALL_DIR="/var/cache/kubernetes-install" mkdir -p ${INSTALL_DIR}

print "readonly {var}={value}".format(var = k, value = pipes.quote(str(v))) ''' < "${kube_env_yaml}") <del> # Infer master status from presence in node pool if [[ $(hostname) = ${NODE_INSTANCE_PREFIX}* ]]; then KUBERNETES_MASTER="false" else </del> <ins> # Infer master status from hostname if [[ $(hostname) == "${INSTANCE_PREFIX}-master" ]]; then </ins> KUBERNETES_MASTER="true" <ins> else KUBERNETES_MASTER="false" </ins> fi }

if [[ -z "${is_push}" ]]; then echo "== kube-up node config starting ==" set-broken-motd <ins> ensure-basic-networking </ins> ensure-install-dir set-kube-env [[ "${KUBERNETES_MASTER}" == "true" ]] && mount-master-pd

echo "== kube-up node config done ==" else echo "== kube-push node config starting ==" <ins> ensure-basic-networking </ins> ensure-install-dir set-kube-env create-salt-pillar