Change:
What happened: The cluster master in a Kubernetes v1.14.10 cluster was upgraded to V1.16.7. The process of upgrading nodes is under way and has been spread over a week so far. The cluster owner reported seeing a high percentage of pod status PATCH failures. The apiserver logs show errors like: The errors are all from the v1.14 nodes and have all been for pods associated with DaemonSets. We don't yet know why pods' IPs are changing, but it appears that pod status patches that include a changed result in an object where contains the old IP, and has the new IP. We believe this is because was introduced in Kubernetes v1.15 and the patch request from a v1.14 node results in both and being present in the result. If you aren't monitoring for request failures it may not obvious that any error is occurring: The pods continue to show their last successfully reported state. Deleting such pods, however, results in the pod being stuck in a state. We are only seeing this for DaemonSet pods - I suspect because whatever resulted in pod IPs changing also causes non-DaemonSet pods to be deleted and recreated. We have a number of cluster still on v1.14 that we expect to upgrade to v1.15 or v1.16. What you expected to happen: Kubernetes v1.14 nodes should be able to update pod status during the transition from v1.14 to v1.16 even if pod IPs change during that time. How to reproduce it (as minimally and precisely as possible): We have recreated this in the IBM Cloud Kubernetes Service by "reloading" nodes. The reload process leaves the Kubernetes node object in place, and essentially reloads the image with the proper kubeconfig to rejoin the cluster. The cluster I was debugging does not appear to have had nodes "reloaded" - something else is triggering the pod IP changes. Create a v1.14 cluster Deploy a DaemonSet (anything with pod networking should suffice) Upgrade cluster master to v1.16 Reload a v1.14 node. One the node is back up, shows the pod's network IP has changed. Kubelet and apiserver logs show the errors describe earlier. Anything else we need to know?: Environment: Kubernetes version (use ): v1.14.x (nodes) and v1.16.7 (master) Cloud provider or hardware configuration: IBM Cloud Kubernetes Service OS (e.g: ): 18.04.4 LTS Kernel (e.g. ): 4.15.0-76-generic Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig api-machinery
/sig network
cc
/priority critical-urgent if it's possible for podIPs to change and the server defaults to , then the API server has to treat the singular pod IP as authoritative if there's a mismatch or the patch case breaks
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry. ‍
/remove-sig api-machinery

Discussion:
return err } <del> // If both fields (v1.PodIPs and v1.PodIP) are provided, then test v1.PodIP == v1.PodIPs[0] </del> <ins> // If both fields (v1.PodIPs and v1.PodIP) are provided and differ, then PodIP is authoritative for compatibility with older kubelets </ins> if (len(in.PodIP) > 0 && len(in.PodIPs) > 0) && (in.PodIP != in.PodIPs[0].IP) { <del> return fmt.Errorf("conversion Error: v1.PodIP(%v) != v1.PodIPs[0](%v)", in.PodIP, in.PodIPs[0].IP) </del> <ins> out.PodIPs = []core.PodIP{ { IP: in.PodIP, }, } </ins> } // at the this point, autoConvert copied v1.PodIPs -> core.PodIPs // if v1.PodIPs was empty but v1.PodIP is not, then set core.PodIPs[0] with v1.PodIP

if err := autoConvert_v1_NodeSpec_To_core_NodeSpec(in, out, s); err != nil { return err } <del> // If both fields (v1.PodCIDRs and v1.PodCIDR) are provided, then test v1.PodCIDR == v1.PodCIDRs[0] </del> <ins> // If both fields (v1.PodCIDRs and v1.PodCIDR) are provided and differ, then PodCIDR is authoritative for compatibility with older clients </ins> if (len(in.PodCIDR) > 0 && len(in.PodCIDRs) > 0) && (in.PodCIDR != in.PodCIDRs[0]) { <del> return fmt.Errorf("conversion Error: v1.PodCIDR(%v) != v1.CIDRs[0](%v)", in.PodCIDR, in.PodCIDRs[0]) </del> <ins> out.PodCIDRs = []string{in.PodCIDR} </ins> } // at the this point, autoConvert copied v1.PodCIDRs -> core.PodCIDRs

} } func Test_v1_PodStatus_to_core_PodStatus(t *testing.T) { <del> // fail v1FailTestInputs := []v1.PodStatus{ </del> <ins> asymmetricInputs := []struct { name string in   v1.PodStatus out  core.PodStatus }{ </ins> { <del> PodIP: "1.1.2.1", // fail becaue PodIP != PodIPs[0] PodIPs: []v1.PodIP{ {IP: "1.1.1.1"}, {IP: "2.2.2.2"}, </del> <ins> name: "mismatched podIP", in: v1.PodStatus{ PodIP: "1.1.2.1", // Older field takes precedence for compatibility with patch by older clients PodIPs: []v1.PodIP{ {IP: "1.1.1.1"}, {IP: "2.2.2.2"}, }, }, out: core.PodStatus{ PodIPs: []core.PodIP{ {IP: "1.1.2.1"}, }, }, }, { name: "matching podIP", in: v1.PodStatus{ PodIP: "1.1.1.1", PodIPs: []v1.PodIP{ {IP: "1.1.1.1"}, {IP: "2.2.2.2"}, }, }, out: core.PodStatus{ PodIPs: []core.PodIP{ {IP: "1.1.1.1"}, {IP: "2.2.2.2"}, }, }, }, { name: "empty podIP", in: v1.PodStatus{ PodIP: "", PodIPs: []v1.PodIP{ {IP: "1.1.1.1"}, {IP: "2.2.2.2"}, }, }, out: core.PodStatus{ PodIPs: []core.PodIP{ {IP: "1.1.1.1"}, {IP: "2.2.2.2"}, }, </ins> }, }, } // success v1TestInputs := []v1.PodStatus{ // only Primary IP Provided

}, }, } <del> // run failed cases for i, failedTest := range v1FailTestInputs { </del> <ins> // run asymmetric cases for _, tc := range asymmetricInputs { testInput := tc.in </ins> corePodStatus := core.PodStatus{} // convert.. <del> if err := corev1.Convert_v1_PodStatus_To_core_PodStatus(&failedTest, &corePodStatus, nil); err == nil { t.Errorf("%v: Convert v1.PodStatus to core.PodStatus should have failed for input %+v", i, failedTest) </del> <ins> if err := corev1.Convert_v1_PodStatus_To_core_PodStatus(&testInput, &corePodStatus, nil); err != nil { t.Errorf("%s: Convert v1.PodStatus to core.PodStatus failed with error:%v for input %+v", tc.name, err.Error(), testInput) } if !reflect.DeepEqual(corePodStatus, tc.out) { t.Errorf("%s: expected %#v, got %#v", tc.name, tc.out.PodIPs, corePodStatus.PodIPs) </ins> } }

} func Test_v1_NodeSpec_to_core_NodeSpec(t *testing.T) { <del> failInputs := []v1.NodeSpec{ { // fail PodCIDRs[0] != PodCIDR PodCIDR:  "10.0.0.0/24", PodCIDRs: []string{"10.0.1.0/24", "ace:cab:deca::/8"}, </del> <ins> asymmetricInputs := []struct { name string in   v1.NodeSpec out  core.NodeSpec }{ { name: "mismatched podCIDR", in: v1.NodeSpec{ PodCIDR:  "10.0.0.0/24", PodCIDRs: []string{"10.0.1.0/24", "ace:cab:deca::/8"}, }, out: core.NodeSpec{ PodCIDRs: []string{"10.0.0.0/24"}, }, }, { name: "unset podCIDR", in: v1.NodeSpec{ PodCIDR:  "", PodCIDRs: []string{"10.0.1.0/24", "ace:cab:deca::/8"}, }, out: core.NodeSpec{ PodCIDRs: []string{"10.0.1.0/24", "ace:cab:deca::/8"}, }, }, { name: "matching podCIDR", in: v1.NodeSpec{ PodCIDR:  "10.0.1.0/24", PodCIDRs: []string{"10.0.1.0/24", "ace:cab:deca::/8"}, }, out: core.NodeSpec{ PodCIDRs: []string{"10.0.1.0/24", "ace:cab:deca::/8"}, }, </ins> }, }

}, } <del> // fail cases for i, failInput := range failInputs { </del> <ins> // run asymmetric cases for _, tc := range asymmetricInputs { testInput := tc.in </ins> coreNodeSpec := core.NodeSpec{} <del> if err := corev1.Convert_v1_NodeSpec_To_core_NodeSpec(&failInput, &coreNodeSpec, nil); err == nil { t.Errorf("%v: Convert v1.NodeSpec to core.NodeSpec failed. Expected an error when coreNodeSpec.PodCIDR != coreNodeSpec.PodCIDRs[0]", i) </del> <ins> // convert.. if err := corev1.Convert_v1_NodeSpec_To_core_NodeSpec(&testInput, &coreNodeSpec, nil); err != nil { t.Errorf("%s: Convert v1.NodeSpec to core.NodeSpec failed with error:%v for input %+v", tc.name, err.Error(), testInput) } if !reflect.DeepEqual(coreNodeSpec, tc.out) { t.Errorf("%s: expected %#v, got %#v", tc.name, tc.out.PodCIDRs, coreNodeSpec.PodCIDRs) </ins> } }