Change:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: of a static pod (really the mirror pod) results in multiple attempts to re-create the mirror pod before it finally succeeds (note that in some runs it goes into Running or Terminating state -- it just depends upon the exact timing of the . Note that in the below trace the pod name is elided from the output. Note that the actual static pod continues to run as normal. What you expected to happen: Mirror pod should start without additional state transitions How to reproduce it (as minimally and precisely as possible): Create a static pod, it, and Anything else we need to know?: Environment: Kubernetes version (use ): 1.14/OpenShift 4.2 (master has the same code) Cloud provider or hardware configuration: AWS (not relevant) OS (e.g: ): Not relevant Kernel (e.g. ): Not relevant Install tools: Not relevant Network plugin and version (if this is a network-related bug): N/A Others:
/sig node
More details: The bug appears to be a race condition between deleting the old mirror pod, the API server deleting its record of it, and creating a replacement that's not related to skew. I suspect the reason I didn't see it bouncing back and forth in my earlier tests was simply that I wasn't running the kubectl get node frequently enough; if I run it continuously, I see it bounce. The sequence is: 1) Somebody kubectl deletes the mirror pod. 2) The static pod code in (around line 1630) notices that the pod has been marked deleted (its deletion timestamp is not nil and it is a mirror pod), deletes it (in DeleteMirrorPod in , via the apiserverClient), and creates a new one (again, via the apiserverClient). 3) The next time around, the static pod code may see either the old pod (that has been deleted) or the new one as the mirror pod. If the old pod is the one that's seen, it sees that there's a deletion timestamp, so it deletes it again -- which, since pod deletion is by name (UID isn't involved) results in the new pod being deleted and yet another new pod being created. If the new pod is the one that's seen, then we break out with the spin cycle complete. Still TBD is why the mirror pod isn't getting replaced right away. But here are traces of the relevant sequences: Bad sequence: Good sequence: The messages at lines 7 and 18 were printed immediately after the call to apiServerClient...Create(). But the key is at lines 9 and 19. At line 9, note that the UID (...) is the UID of the old mirror pod, which has a deletion timestamp and therefore the mirror code deletes the pod -- but the pod that's actually in existence at the time is the new pod. At line 19, however, note that the mirror pod is now the new one, which is not marked for deletion, and is therefore left alone. The solution (I'll open a PR shortly) is to check both the pod name and the UID to make sure that we're in fact trying to delete the pod we think we want to delete.
Thanks for the detailed issue :) Really awesome!
I'm seeing this issue on an upgrade from v1.14.4 to v1.15.1 following the on a CentOS 7 system. After installing the updated kubelet package and restarting kubelet per the instructions, the cluster went into a loop of continually restarting the etcd pod from one control plane node and the scheduler pod from another control plane node. This continued for approximately 10 minutes before the etcd pod finally stayed up. It was another 5 minutes before the scheduler pod finally ran. I noticed v1.15.2 was released although the change log doesn't show this issue included. Was there a plan to back-port this fix to 1.15 and 1.14 (not to the same extent, but I have seen this behavior on a 1.14.4 cluster)? Since this behavior can happen to the apiserver pod, it has the potential to be very disruptive to a cluster.
The actual etcd pod, or the mirror pod (which is more likely)? I hadn't specifically planned on backporting it, but it shouldn't be a problem; this code has been stable.
Ah, good point--the actual pod does appear to remain running. That makes it less important than I was initially thinking. However, it is a bit disconcerting for someone who happens to be looking at the state of the cluster (or for monitoring) to see the pod appear to be stuck in a restart loop. If it were just a few iterations, it might not matter too much. However, I'm seeing this happen for minutes at a time. For example, to verify that the real pod remained available I did a on the apiserver pod for one master, and output showed it in the restart loop for 15 minutes before it settled down. IMO that's worth backporting if it's low risk.
I opened a cherrypick against 1.15 for this.
Appreciated. Thanks!

Discussion:
if mirrorPod.DeletionTimestamp != nil || !kl.podManager.IsMirrorPodOf(mirrorPod, pod) { // The mirror pod is semantically different from the static pod. Remove // it. The mirror pod will get recreated later. <del> klog.Warningf("Deleting mirror pod %q because it is outdated", format.Pod(mirrorPod)) if err := kl.podManager.DeleteMirrorPod(podFullName); err != nil { </del> <ins> klog.Infof("Trying to delete pod %s %v", podFullName, mirrorPod.ObjectMeta.UID) var err error deleted, err = kl.podManager.DeleteMirrorPod(podFullName, &mirrorPod.ObjectMeta.UID) if deleted { klog.Warningf("Deleted mirror pod %q because it is outdated", format.Pod(mirrorPod)) } else if err != nil { </ins> klog.Errorf("Failed deleting mirror pod %q: %v", format.Pod(mirrorPod), err) <del> } else { deleted = true </del> } } }

"k8s.io/api/core/v1" "k8s.io/apimachinery/pkg/api/errors" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" <ins> "k8s.io/apimachinery/pkg/types" </ins> clientset "k8s.io/client-go/kubernetes" "k8s.io/klog" kubecontainer "k8s.io/kubernetes/pkg/kubelet/container"

CreateMirrorPod(pod *v1.Pod) error // DeleteMirrorPod deletes the mirror pod with the given full name from // the API server or returns an error. <del> DeleteMirrorPod(podFullName string) error </del> <ins> DeleteMirrorPod(podFullName string, uid *types.UID) (bool, error) </ins> } // basicMirrorClient is a functional MirrorClient.  Mirror pods are stored in

return err } <del> func (mc *basicMirrorClient) DeleteMirrorPod(podFullName string) error { </del> <ins> // DeleteMirrorPod deletes a mirror pod. // It takes the full name of the pod and optionally a UID.  If the UID // is non-nil, the pod is deleted only if its UID matches the supplied UID. // It returns whether the pod was actually deleted, and any error returned // while parsing the name of the pod. // Non-existence of the pod or UID mismatch is not treated as an error; the // routine simply returns false in that case. func (mc *basicMirrorClient) DeleteMirrorPod(podFullName string, uid *types.UID) (bool, error) { </ins> if mc.apiserverClient == nil { <del> return nil </del> <ins> return false, nil </ins> } name, namespace, err := kubecontainer.ParsePodFullName(podFullName) if err != nil { klog.Errorf("Failed to parse a pod full name %q", podFullName) <del> return err </del> <ins> return false, err </ins> } <del> klog.V(2).Infof("Deleting a mirror pod %q", podFullName) // TODO(random-liu): Delete the mirror pod with uid precondition in mirror pod manager if err := mc.apiserverClient.CoreV1().Pods(namespace).Delete(name, metav1.NewDeleteOptions(0)); err != nil && !errors.IsNotFound(err) { klog.Errorf("Failed deleting a mirror pod %q: %v", podFullName, err) </del> <ins> klog.V(2).Infof("Deleting a mirror pod %q (uid %#v)", podFullName, uid) var GracePeriodSeconds int64 GracePeriodSeconds = 0 if err := mc.apiserverClient.CoreV1().Pods(namespace).Delete(name, &metav1.DeleteOptions{GracePeriodSeconds: &GracePeriodSeconds, Preconditions: &metav1.Preconditions{UID: uid}}); err != nil { // Unfortunately, there's no generic error for failing a precondition if !(errors.IsNotFound(err) || errors.IsConflict(err)) { // We should return the error here, but historically this routine does // not return an error unless it can't parse the pod name klog.Errorf("Failed deleting a mirror pod %q: %v", podFullName, err) } return false, nil </ins> } <del> return nil </del> <ins> return true, nil </ins> } // IsStaticPod returns true if the pod is a static pod.

func (pm *basicManager) DeleteOrphanedMirrorPods() { podFullNames := pm.getOrphanedMirrorPodNames() for _, podFullName := range podFullNames { <del> pm.MirrorClient.DeleteMirrorPod(podFullName) </del> <ins> pm.MirrorClient.DeleteMirrorPod(podFullName, nil) </ins> } }

"sync" "k8s.io/api/core/v1" <ins> "k8s.io/apimachinery/pkg/types" </ins> "k8s.io/apimachinery/pkg/util/sets" cp "k8s.io/kubernetes/pkg/kubelet/checkpoint" "k8s.io/kubernetes/pkg/kubelet/checkpointmanager"

return nil } <del> func (fmc *FakeMirrorClient) DeleteMirrorPod(podFullName string) error { </del> <ins> // TODO (Robert Krawitz): Implement UID checking func (fmc *FakeMirrorClient) DeleteMirrorPod(podFullName string, _ *types.UID) (bool, error) { </ins> fmc.mirrorPodLock.Lock() defer fmc.mirrorPodLock.Unlock() fmc.mirrorPods.Delete(podFullName) fmc.deleteCounts[podFullName]++ <del> return nil </del> <ins> return true, nil </ins> } func (fmc *FakeMirrorClient) HasPod(podFullName string) bool {

} // DeleteMirrorPod provides a mock function with given fields: podFullName <del> func (_m *MockManager) DeleteMirrorPod(podFullName string) error { </del> <ins> func (_m *MockManager) DeleteMirrorPod(podFullName string, _ *types.UID) (bool, error) { </ins> ret := _m.Called(podFullName) var r0 error if rf, ok := ret.Get(0).(func(string) error); ok { <del> r0 = rf(podFullName) </del> <ins> return true, rf(podFullName) </ins> } else { r0 = ret.Error(0) } <del> return r0 </del> <ins> return false, r0 </ins> } // DeleteOrphanedMirrorPods provides a mock function with given fields: