Change:
When looking at Kubelet healthz through proxy on master I'm getting: cc
cc
kubelet expects to be the hostname, while master uses IP in the request instead. /cc people who knows more about proxy:
Master's expectation looks wrong.
The idea of that health check is to ensure that the master schedules work to the name expected by the kubelet. Otherwise sometimes you end up in a situation where the master is scheduling work for IP and the kubelet is looking for work for 'hostname -f'; the node status says 'Ready' but no pods get started on that node. User's must use --hostname_override to correct the situation. It has been a while since I have looked at that code so things may have changed, but in this case is the master scheduling using the hostname "10.240.170.229" ? If so then pods will not start on that node. If not then the health check needs a better way of getting the hostname as the master knows it.
I don't think this is related to the proxy, I think your master has an incorrect node configuration, as says.
- I checked this on and clusters as well, with the same result. All of us were running default GCE configuration.
That's very weird, I reproduce, too. But it has nothing to do with the proxy, rather it's the requested host (????!?): So kubelet gives different answers depending on whether you refer to it as "localhost" or by its IP address.
If kubelet is doing the healthz check about the host in the HTTP request, then that's wrong. To be useful, it should be making an assertion about the Node object that represents it in the system.
To give more context, the change was introduced in I think was describing a situation where kubelet and master have discrepancy on what the node is. E.g., kubelet thinks it's node HOSTNAME, while master thinks the node is IP. In this case, kubelet wouldn't know which Node object to check. if I understand correctly, this should not happen now because kubelet would not update the node status of the incorrect node, so the node would eventually become unavailable kubelet can register itself now
Yeah the original patch in sent the host name presumed by master along with the request but we though the hostname from the header sufficed as we could not come up with a scenario where the two differ. That can be changed back if needed. It looks like the localhost bypass was later in in your cluster if you disable the health check do pods actually start on the node ?
hmm... do you mean the master would not be able to reach the node if it has the incorrect name ? The hostname is correct it is just different from what the node uses. Say for example the host uses "" and the node node knows itself as "93.184.216.34". The health check will pass because the node is reachable, but pods don't start because work no work is scheduled for "93.184.216.34" Yeah I think that is a safe path.
kubelet updates node status now, and if the node status isn't updated for sometime, node controller would deem the node unhealthy. E.g., if the node knows itself as "93.184.216.34", it'd try to update the status of node "93.184.216.34". The status of node "" would not be updated by anyone, and would eventually become unhealthy.
It fails through the proxy because master's proxy code uses IP address (not hostname) to reach the node. Master is actually doing the right thing there. Master doesn't directly health check kubelets any more (monit does) so that's why this isn't currently breaking anything. I'm guessing the localhost exception was because that's how monit health checks; but that's evidence that the original assumption that one could use the HTTP request's host for this was faulty, and the check should be completely removed or done some other way.
got it! I see. The check is no longer correct then and given what explained the check is unnecessary :) I'll open a PR to remove it.
thanks!

Discussion:
healthz.InstallHandler(s.restfulCont, healthz.PingHealthz, healthz.NamedCheck("docker", s.dockerHealthCheck), <del> healthz.NamedCheck("hostname", s.hostnameHealthCheck), </del> healthz.NamedCheck("syncloop", s.syncLoopHealthCheck), ) var ws *restful.WebService

return nil } <del> func (s *Server) hostnameHealthCheck(req *http.Request) error { masterHostname, _, err := net.SplitHostPort(req.Host) if err != nil { if !strings.Contains(req.Host, ":") { masterHostname = req.Host } else { return fmt.Errorf("Could not parse hostname from http request: %v", err) } } // Check that the hostname known by the master matches the hostname // the kubelet knows hostname := s.host.GetHostname() if masterHostname != hostname && masterHostname != "127.0.0.1" && masterHostname != "localhost" { return fmt.Errorf("Kubelet hostname "%v" does not match the hostname expected by the master "%v"", hostname, masterHostname) } return nil } </del> // Checks if kubelet's sync loop  that updates containers is working. func (s *Server) syncLoopHealthCheck(req *http.Request) error { duration := s.host.ResyncInterval() * 2