Change:
During kube-proxy bootstrap, kube-proxy retrieves node IP by accessing node object. Because kube-proxy is a static pod, kubelet may start kube-proxy before completing self registration. Hence kube-proxy may start up without knowing the correct node IP.
to confirm.
I'm hitting this as well, for the same reason - kube-proxy comes up before node registration is complete. I'm wondering what are the consequences of this?
I completely missed the issue :- kube-proxy uses the host network, so it's not blocked by kubelet registering and getting the pod CIDR. Can't kube-proxy wait and/or retry? is the bug still valid or has it been fixed?
This is not fixed. We are moving kube-proxy to daemonset. I am wondering if there is some easy config to make sure kube-proxy starts up after node status becomes ready or some sort.
yay! Is making kube-proxy blocked until node is ready not an option?
/assign
GitHub didn't allow me to assign the following users: MikeSpreitzer. Note that only can be assigned. <detailsIn response to : /assign </details
/assign
Update?
I got derailed by ; clues there would be nice. I will get back to this today, one way or another.
I have been reading about tests, wondering where to add a test case that covers this problem. It is not clear to me that Kubernetes has a test suite that can test race conditions in cluster creation.
There are some reboot/restart test.
FYI:
A test case that probes this issue would be even more intimate with the way the cluster is deployed, needing to inject a controlled delay in the node startup sequence.
Will this make 1.7?
move to the next milestone
BTW, always makes the kube-proxy before the kubelet, so it is easy to probe this issue in manual testing.
Next milestone reason: it seems that there are still some conversations about this the PR fix around whether or not we want to make require the presence of . It seems warranted then to accept this fix in the next release.
We need to address this in a patch release, but it should not block 1.7.
This is impacting one of our worker nodes as well, which was working fine and then after a restart, the API server communication was impacted (for POST requests, e.g. , ). We noticed the the error in the kube-proxy logs. The workaround is to detect the failure via the logs, and restart the kube-proxy pod.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten /remove-lifecycle stale
As far as I can tell, the nodeIP is used in one particular case and nowhere else. Specifically a service of type=LoadBalancer, which also uses LoadBalancerSourceRanges field AND specifies the node itself as a valid source range for the LB. That's a pretty corner case. I bet we could find a better way to express that, and eliminate that parameter and error message entirely. I doubt it is actually causing anyone problems unless I totally misunderstand what that field is doing
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Since the PR is merged to master, is also used in . Now if this race condition happens, kube-proxy is not able to determine the correct nodeIP. This affects all the nodeport services with kube-proxy mode ipvs. Please reopening the issue.
You can't reopen an issue/PR unless you authored it or you are a collaborator. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
looks like this shouldn't have gotten closed anyway /reopen /remove-lifecycle rotten /lifecycle frozen
Reopened this issue. <detailsIn response to : looks like this shouldn't have gotten closed anyway /reopen /remove-lifecycle rotten /lifecycle frozen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Can't we use the similar fix given in PR .

Discussion:
nodeIP := net.ParseIP(config.BindAddress) if nodeIP.IsUnspecified() { nodeIP = utilnode.GetNodeIP(client, hostname) <ins> if nodeIP == nil { return nil, fmt.Errorf("unable to get node IP for hostname %s", hostname) } </ins> } if proxyMode == proxyModeIPTables { klog.V(0).Info("Using iptables Proxier.")

"//staging/src/k8s.io/apimachinery/pkg/apis/meta/v1:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/types:go_default_library", "//staging/src/k8s.io/apimachinery/pkg/util/strategicpatch:go_default_library", <ins> "//staging/src/k8s.io/apimachinery/pkg/util/wait:go_default_library", </ins> "//staging/src/k8s.io/client-go/kubernetes:go_default_library", "//staging/src/k8s.io/client-go/kubernetes/typed/core/v1:go_default_library", "//vendor/k8s.io/klog:go_default_library",

metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/types" "k8s.io/apimachinery/pkg/util/strategicpatch" <ins> "k8s.io/apimachinery/pkg/util/wait" </ins> clientset "k8s.io/client-go/kubernetes" v1core "k8s.io/client-go/kubernetes/typed/core/v1" )

} // GetNodeIP returns the ip of node with the provided hostname <ins> // If required, wait for the node to be defined. </ins> func GetNodeIP(client clientset.Interface, hostname string) net.IP { var nodeIP net.IP <del> node, err := client.CoreV1().Nodes().Get(hostname, metav1.GetOptions{}) if err != nil { klog.Warningf("Failed to retrieve node info: %v", err) return nil </del> <ins> backoff := wait.Backoff{ Steps:    5, Duration: 1 * time.Second, Factor:   2.0, Jitter:   0.2, </ins> } <del> nodeIP, err = GetNodeHostIP(node) if err != nil { klog.Warningf("Failed to retrieve node IP: %v", err) return nil </del> <ins> err := wait.ExponentialBackoff(backoff, func() (bool, error) { node, err := client.CoreV1().Nodes().Get(hostname, metav1.GetOptions{}) if err != nil { klog.Errorf("Failed to retrieve node info: %v", err) return false, nil } nodeIP, err = GetNodeHostIP(node) if err != nil { klog.Errorf("Failed to retrieve node IP: %v", err) return false, err } return true, nil }) if err == nil { klog.Infof("Successfully retrieved node IP: %v", nodeIP) </ins> } return nodeIP }