Change:
The kubelet logic used to get the Node addresses when using an external provider, temporary assigns an address discovered from the node , this address can be overriden later by the external cloud provider This was implemented in to improve the bootstrapping problems with some external cloud providers The side effect of this behavior is that if the external cloud provider sets a different address/es, then the pods with will be showing the "first" addresses as their , since we don't mutate those IPs once set to avoid problems I think that we should revert the existing logic for external cloud providers, since is causing a temporary and inconsistent state and it seem to address mainly a problem with kubeadm but kubeadm folks seems to not need this functionality Announced mailing list /kind bug /sig network /sig node
/assign /cc
So we don't want the external provider to update the IP of the node, right? But I think even if it updates the address, for pods with hostnetwork, it still gets the first address from nodeStatus as podIPs, why is it inconsistent?
the contrary, we don't want to shortcut the address assignment, if cloud provider is external the node should wait for the cloud provider to assign the address. Let me put an example, a node with IPs 10.0.0.1 and 2001:db2::3 and cloud-provider external that sets the IPv6 address first. Right, now, the Node will use prefer IPv4 when it starts and assign the IP 10.0.0.1 to the node, hostNetwork pods will get that IP. and later the cloud provider will override the IPs and sets 2001:db2::3 as the address, so new hostNetwork pods will use the IPv6 address but old hostNetwork pods will have 10.0.0.1
I see, Now I think it makes sense, but we need to confirm that kubeadm really doesn't need it anymore.
^^^
I don't know the history of why it works the way it currently does, but I agree that it's weird and seems likely to lead to problems, so if it turns out it's not necessary, then let's change it.
This is why we taint nodes initially and only remove the taint once cloud provider has initialized the node. So this is reallly an issue for pods tolerating this taint
i don't understand what is being asked exactly, but if the Q is again whether kubeadm supports self-hosted control plane components in a DS, then no. kubeadm does support the --cloud-privider=external on paper, but we don't have tests. i don't think we want to maintain them either. do you have context?
For current behavior, the workaround would be if I understand correctly. I don't know if there are any cloud provider installers that depend on this behavior. I prefer to fix the weird behavior to avoid potential problems. Or a problem detector has to cover this to check if a hostNet pod is not using the correct node IP.
yeah, but I think that is fair to give those pods the responsibility to deal with the known behavior (node will not have addresses until cloud provider external initializes) , than make the kubelet have two different addresses during its lifetime (that we discussed extensively and ) We discarded this behavior because most components, apps and features like downward API depend on the existing behavior of not changing the original IP For the temporal status without addresses we can just set the node as notReady
NetworkUnavailable might be more appropriate, but overall makes sense I think
is there a reported issue or bug that is reproducible based on what you describe in the isuse description? Logically what you said makes sense but I'm wondering how common the problem actually is
Ah, is it just or are there others?
Oh -- is not even related, because they are not setting --cloud-provider=external
yeah, I opened this issue because when I tried to bring up an IPv6 only cluster with kops in GCE it failed and one of the problems was that the hostNetwork pods were using IPv4 as pod ips ... is not likely anybody hits this with IPv4 or dual stack, since cloud-provider and kubelet will agree most of the times, but as soon that you use IPv6 only or IPv6 primary you ends with this problem
Isn't this a configuration issue on the kubelet and not with external cloud providers then? You should configure kubelet to bootstrap initially with an IPv6 address and not IPv4.
Still does not work for dual-stack, or if there are multiple addresses on the node, or multiple interfaces .... I don't see the reason why kubelet should configure the node addresses in the first place just to wait for the cloud-provider to override that ... the pods bootstrap problem is another thing, kubelet should not be ready if there are no addresses in the node , but current behavior is compensating the problem of having an external source for the address allocation, that is clearly not generalizable
yeah, I opened this issue because when I tried to bring up an IPv6 only cluster with kops in GCE it failed and one of the problems was that the hostNetwork pods were using IPv4 as pod ips ... is not likely anybody hits this with IPv4 or dual stack, since cloud-provider and kubelet will agree most of the times, but as soon that you use IPv6 only or IPv6 primary you ends with this problem For cloud-provider-vsphere, what we did to workaround is to identify the preferred ip address to get, which interface, which rank, then in bootstrap time we get it and pass it to kubelet flag . Now support dual stack address with external cloud-provider. Without the kubelet starting node with some IP, if there is issue in CPI code or in the config, then it would be hard for us to debug CPI since getting the log will getting error that node doesn't have prefered ip. Might need to get the container log from containerd directly by ssh into the node. You mentioned this is for GCE, I remember brought it up in here that now the is also if CCM has route controller enable, which doesn't have clear ownership between CMM and CNI which should reconcile it. And according to the CCM logic , in GCE case it will managed the in CCM, and probably will mark it as false once the route is ready, then node could possibly becomes ready. So maybe that's why in your case the node becomes ready even though the node doesn't have the address you preferred.
I found it trying with GCE, but that is not important, I can hit this with Openshift or Rancher or Kind ... my goal is to define a general behavior of the kubelet that is consistent and easier to think about, without breaking anyone of course, but current implementation is a footgun and seems tailor made... do we enforce all cloud providers to pass the --node-ip flag to kubelet to match their IPs? I will put this for the next sig-network agenda so we can go deeper and talk more about it
Another example of this problem in Azure and CAPG
Yeah, the NetworkUnavailable condition/taint is not extensible or reusable. Any change to its current semantics will not work correctly in some environments.
Ohk we ran into similar issue today, We didn't pass --node-ips to kubelet, and we are setting ipv6 as primary ipfamily, but kubelet chose ipv4 address at the beginning, so all hostNetwork pod has ipv4 address. Now even after CPI has fixed the node ip to put ipv6 address as primary address, all following pods still has ipv4 addresses regardless. Example: Node addresses:
so , does it mean you agree we need to fix this as in
I'm in favor of removing the behavior to bootstrap initial addresses as long as we are confident it is not breaking some exitsing behavior that some provider is depending on. I still find the scenario in a little odd, those pods should not be running on the node if the CCM has no initialized them yet. I wonder if the actual problem here is that node addresses are only updated after the first initialization: Which means pods can be admitted between the time time the cloud provider taint is removed and when addresses are updated.
so , does it mean you agree we need to fix this as in At least from cloud-provider-vsphere it shouldn't depend on this behavior to bootstrap. It's just that it would be harder for people to debug the CPI issue or issues in other component that comes before CPI like KCM/kube-apiserver/etcd in the future if something goes wrong before CPI could initialize the node. Example issue: I still find the scenario in a little odd, those pods should not be running on the node if the CCM has no initialized them yet. Yeah this is weird. For pod like apiserver pod has ipv4 address, which make sense since it comes before CPI. Kubelet could choose the wrong IP at the beginning now CPI already removes the taints and updated the node with correct IP address, also the node has correct podCIDRs, with ipv6 address at the front. Still if I create a new pod on that node, it will be assigned with a ipv4 address as primary ip. For example that test pod was created long after the CPI initialized the node. And we only experience issue with pod that get created on this node, all other pod on other node will have ipv6 address as primary address.
/triage accepted /priority important-longterm
that should not be a problem if having node addresses is a requirement of node readiness
The AWS EKS AMIs that we provide specify which avoids this issue. If that flag is not passed, then the same problem exists along with a slightly worse one that we intend to fix,
good catch is event more complex, since the ccm will not add addresses meanwhile the node is tainted ‍

Discussion:
if externalCloudProvider { // If --cloud-provider=external and node address is already set, // then we return early because provider set addresses should take precedence. <del> // Otherwise, we try to look up the node IP and let the cloud provider override it later </del> <ins> // Otherwise, we try to use the node IP defined via flags and let the cloud provider override it later </ins> // This should alleviate a lot of the bootstrapping issues with out-of-tree providers if len(node.Status.Addresses) > 0 { return nil } <ins> // If nodeIPs are not specified wait for the external cloud-provider to set the node addresses. // Otherwise uses them on the assumption that the installer/administrator has the previous knowledge // required to ensure the external cloud provider will use the same addresses to avoid the issues explained // in https://github.com/kubernetes/kubernetes/issues/120720. // We are already hinting the external cloud provider via the annotation AnnotationAlphaProvidedIPAddr. if !nodeIPSpecified { return nil } </ins> } if cloud != nil { cloudNodeAddresses, err := nodeAddressesFunc()

shouldError:      false, }, { <del> name:              "cloud provider is external", </del> <ins> name:              "cloud provider is external and nodeIP specified", </ins> nodeIP:            netutils.ParseIPSloppy("10.0.0.1"), nodeAddresses:     []v1.NodeAddress{}, cloudProviderType: cloudProviderExternal,

shouldError: false, }, { <ins> name:              "cloud provider is external and nodeIP unspecified", nodeIP:            netutils.ParseIPSloppy("::"), nodeAddresses:     []v1.NodeAddress{}, cloudProviderType: cloudProviderExternal, expectedAddresses: []v1.NodeAddress{}, shouldError:       false, }, { name:              "cloud provider is external and no nodeIP", nodeAddresses:     []v1.NodeAddress{}, cloudProviderType: cloudProviderExternal, expectedAddresses: []v1.NodeAddress{}, shouldError:       false, }, { </ins> name: "cloud doesn't report hostname, no override, detected hostname mismatch", nodeAddresses: []v1.NodeAddress{ {Type: v1.NodeInternalIP, Address: "10.1.1.1"},