opened
nkinkade
username_0
Just recently I have noticed in sandbox and staging that the first API server in each cluster, after being rebooted, fails to rejoin the API cluster. The error from the kubelet is that it is getting a connection refused from the API load-balancer URL. Wondering if this could be some sort of a hairpin issue in the loadbalancer, I removed the `us-central1-a` (the one failing to rejoin) from the load balancer, and almost immediately it rejoined.

What we have is a situation where backends of the load-balancer are attempting to communicate with the load-balancer itself, a sort of self-referential problem. It would that the when a backend attempts to communicate with the load-balancer of which it is a part, that the traffic is sent back to that same backend/node. This doesn't work because the kubelet needs to communicate with the API cluster _and_ is also responsible for starting up the api-server pod.

We are currently using a "legacy" TCP/UDP target-pool-based external load balancer. Among other things, this type of backend only supports HTTP health checks, which necessitates the use of the `gcp-loadbalancer-proxy`. Perhaps these issues might go away if we move to the more modern "backend service" type of external load-balancer? It seems worth the trouble, if for nothing else than to get rid of the `gcp-loadbalancer-proxy` and let GCP health checks talk directly to the nodes in the backend instance group using HTTPS.
API nodes sometimes fail to rejoin the API cluster
issue
closed
nkinkade
username_0
issue