Change:
What happened: We have installed k8s cluster on openstack cloud provider setup. K8s cluster is dual-stack enabled. During fresh deployment or worker node reboots (master nodes , worker nodes) , k8s control plane pods (other host networking pods aswell) such as kube-proxy, kube-scheduler , kube-controller-manager get only 1 IP address instead of dual-stack address. What you expected to happen: We expect all host networking pods to get dual-stack addresses , since cluster is dual-stack enabled. How to reproduce it (as minimally and precisely as possible): a) Openstack cloud provider setup with minimum 4-5 VMs. b) 3 master nodes, 2 worker nodes c) Execute simultaneous reboot of all nodes. Environment: Kubernetes version : v1.22.0
/sig node
/sig network /triage needs-information /assign Hi When you describe your node what are the IPs assigned in node object? kubelet copies from node object into pods. The way they are set on node object is by the cloud provider, manual via kubelet's flag or auto-detect which prefers IPv4 (check: ).
when i describe the node object , they contain dual-stack addresses. Yes i debugged kubelet with few additional logs. I could see that when kubelet is reading node obj to create these static pods, it sees only 1 IP. But eventually node object is populated with both the addresses. I also understand, is for baremetal installation. But we have cloud provider setup which might be delaying setting both the IPs in node obj. Before cloud provider sets both the IPs ,kubelet is fast enough to create these static pods. Let me know if you need any additional logs.
thanks again Few additional questions: are these static pods? When the node object gets its dualstack addresses and pods were created after the node object updates do you still get single address pods?
I've chatted with the reporter in slack, it seems a race on populating the node addresses by the cloud-provider,
Yes. It would seem so. Two things here i am trying to clarify. cloud providers node register with taints (no schedule) until cloud provider adds provider-Id and other details including IPs. these are static pods then I am not sure if kubelet respects no-schedule (those are respected by scheduler which is out of the picture for static pods) If they are then they will run - most likely - before node gets IP addresses from cloud provider. The questions i asked above should put some lights on this.
Yes i am referring to kube-proxy , kube-api-server ,kube-scheduler pods which use host-networking. This is not under our control. I mean , we are just rebooting the nodes in quick succession and at the end we just check the IP addresses of the above static pods. As i see the code , kubelet brings them up as soon as he is UP.
IMO the cloud provider really needs to figure out the IPs before kubelet starts. OR the node startup needs to block until both IP addresses are available so that the cloud provider and/or kubelet can read them.
Should we accept this as a bug? Do we think we need to force some taint on new nodes that cloud-providers need to clear (obviously not compatible, but you get the idea)?
-- There is a no-schedule taint already there (if cloud provider is configured on kubelet). It is a bug (or undocumented behavior). I think we should always update PodIPs in host network mode. /triage accepted
xref
so i looked at this again. I believe should fix this issue as well. i don't think we should place this behind a feature gate. We already update when the host IP changes. We just need to edit the release note of to describe the behavior and caveats. Thoughts?
Please ignore that comment. The PR does not solve for
/remove-sig node
: Any more info needed from me ?
no thank you. We are addressing this by and the related KEP.
, any idea when will the fix be available ?

Discussion:
klog.V(4).InfoS("Cannot get host IPs", "err", err) } else { s.HostIP = hostIPs[0].String() <del> if kubecontainer.IsHostNetworkPod(pod) && s.PodIP == "" { s.PodIP = hostIPs[0].String() s.PodIPs = []v1.PodIP{{IP: s.PodIP}} if len(hostIPs) == 2 { </del> <ins> // HostNetwork Pods inherit the node IPs as PodIPs. They are immutable once set, // other than that if the node becomes dual-stack, we add the secondary IP. if kubecontainer.IsHostNetworkPod(pod) { // Primary IP is not set if s.PodIP == "" { s.PodIP = hostIPs[0].String() s.PodIPs = []v1.PodIP{{IP: s.PodIP}} } // Secondary IP is not set #105320 if len(hostIPs) == 2 && len(s.PodIPs) == 1 { </ins> s.PodIPs = append(s.PodIPs, v1.PodIP{IP: hostIPs[1].String()}) } }

criPodIPs: []string{"192.168.0.1"}, podIPs: []v1.PodIP{ {IP: "192.168.0.1"}, <ins> {IP: "fd01::1234"}, </ins> }, }, {

} } <ins> func TestNodeAddressUpdatesGenerateAPIPodStatusHostNetworkPodIPs(t *testing.T) { testcases := []struct { name           string nodeIPs        []string nodeAddresses  []v1.NodeAddress expectedPodIPs []v1.PodIP }{ { name:    "Immutable after update node addresses single-stack", nodeIPs: []string{"10.0.0.1"}, nodeAddresses: []v1.NodeAddress{ {Type: v1.NodeInternalIP, Address: "1.1.1.1"}, }, expectedPodIPs: []v1.PodIP{ {IP: "10.0.0.1"}, }, }, { name:    "Immutable after update node addresses dual-stack - primary address", nodeIPs: []string{"10.0.0.1", "2001:db8::2"}, nodeAddresses: []v1.NodeAddress{ {Type: v1.NodeInternalIP, Address: "1.1.1.1"}, {Type: v1.NodeInternalIP, Address: "2001:db8::2"}, }, expectedPodIPs: []v1.PodIP{ {IP: "10.0.0.1"}, {IP: "2001:db8::2"}, }, }, { name:    "Immutable after update node addresses dual-stack - secondary address", nodeIPs: []string{"10.0.0.1", "2001:db8::2"}, nodeAddresses: []v1.NodeAddress{ {Type: v1.NodeInternalIP, Address: "10.0.0.1"}, {Type: v1.NodeInternalIP, Address: "2001:db8:1:2:3::2"}, }, expectedPodIPs: []v1.PodIP{ {IP: "10.0.0.1"}, {IP: "2001:db8::2"}, }, }, { name:    "Immutable after update node addresses dual-stack - primary and secondary address", nodeIPs: []string{"10.0.0.1", "2001:db8::2"}, nodeAddresses: []v1.NodeAddress{ {Type: v1.NodeInternalIP, Address: "1.1.1.1"}, {Type: v1.NodeInternalIP, Address: "2001:db8:1:2:3::2"}, }, expectedPodIPs: []v1.PodIP{ {IP: "10.0.0.1"}, {IP: "2001:db8::2"}, }, }, { name:    "Update secondary after new secondary address dual-stack", nodeIPs: []string{"10.0.0.1"}, nodeAddresses: []v1.NodeAddress{ {Type: v1.NodeInternalIP, Address: "10.0.0.1"}, {Type: v1.NodeInternalIP, Address: "2001:db8::2"}, }, expectedPodIPs: []v1.PodIP{ {IP: "10.0.0.1"}, {IP: "2001:db8::2"}, }, }, } for _, tc := range testcases { t.Run(tc.name, func(t *testing.T) { testKubelet := newTestKubelet(t, false /* controllerAttachDetachEnabled */) defer testKubelet.Cleanup() kl := testKubelet.kubelet for _, ip := range tc.nodeIPs { kl.nodeIPs = append(kl.nodeIPs, netutils.ParseIPSloppy(ip)) } kl.nodeLister = testNodeLister{nodes: []*v1.Node{ { ObjectMeta: metav1.ObjectMeta{Name: string(kl.nodeName)}, Status: v1.NodeStatus{ Addresses: tc.nodeAddresses, }, }, }} pod := podWithUIDNameNs("12345", "test-pod", "test-namespace") pod.Spec.HostNetwork = true for _, ip := range tc.nodeIPs { pod.Status.PodIPs = append(pod.Status.PodIPs, v1.PodIP{IP: ip}) } if len(pod.Status.PodIPs) > 0 { pod.Status.PodIP = pod.Status.PodIPs[0].IP } // set old status podStatus := &kubecontainer.PodStatus{ ID:        pod.UID, Name:      pod.Name, Namespace: pod.Namespace, } podStatus.IPs = tc.nodeIPs status := kl.generateAPIPodStatus(pod, podStatus) if !reflect.DeepEqual(status.PodIPs, tc.expectedPodIPs) { t.Fatalf("Expected PodIPs %#v, got %#v", tc.expectedPodIPs, status.PodIPs) } if kl.nodeIPs[0].String() != status.PodIPs[0].IP { t.Fatalf("Expected HostIP %q to equal PodIPs[0].IP %q", status.HostIP, status.PodIPs[0].IP) } }) } } </ins> func TestGenerateAPIPodStatusPodIPs(t *testing.T) { testcases := []struct { name      string

} // PodSandboxNetworkStatus is the status of the network for a PodSandbox. <ins> // Currently ignored for pods sharing the host networking namespace. </ins> type PodSandboxNetworkStatus struct { // IP address of the PodSandbox. Ip string `protobuf:"bytes,1,opt,name=ip,proto3" json:"ip,omitempty"`

string ip = 1; } // PodSandboxNetworkStatus is the status of the network for a PodSandbox. <ins> // Currently ignored for pods sharing the host networking namespace. </ins> message PodSandboxNetworkStatus { // IP address of the PodSandbox. string ip = 1;