Change:
cc
PTAL
The root cause is that the node controller and kubelet delete the pods using their names (they should use UID instead): (this is not an exhaustive list) In test-, we create and delete the same pod multiple times, so it's possible that the node controller or the kubelet sends a deletion request to the api server, with the intention of deleting the Nth occurrence of a pod, while it's actually deleting the N+1th occurrence of the pod. To reproduce the flake, I introduced a random delay between 500~1500 ms in the node controller's execution path (), then using a script to create/delete a pod repeatedly, and the flake will occur after several rounds.
Why is it not okay to delete by name?
namespace+name pair is not an unique identifier across time.
the client library and the API server don't support deletion by UID, I think we should support.
Thanks!! My thoughts: does node controller care about these pods? I'm pretty sure the original author wasn't expecting that. a temporary fix, is it hard to just use new names for every incarnation of these pods?
Deleting by UID is probably a bigger can of worms than we want to open right now, although it will eventually be important. Thinking out loud here, but maybe it'd be possible to add a resource version to the delete command. I.e., "delete only if it hasn't changed since rv=xxxxx". But I still think it's probably ideal to turn off the node controller for this test.
I'm also thinking if we can centralize all the pod deletion logic into kubelet, but it seems some of the logic need to stay in the node controller, e.g, , because only the node controller knows if the node exists. deletion probably can be done by the kubelet.
Looks like node controller is getting an update of the pod where DeleteionTimestamp is set, which seems like a bug if --grace-period is set to 0. My understanding is that if grace period is zero, the pod should get immediately deleted. That pod will never make it to a kubelet because it hasn't been scheduled yet. Thus which kubelet would delete it? I'm pretty sure none of those deletions can be handled in the kubelet.
Btw, when running hack/test- (), the node controller generates many messages about the failed deletions: or , these are because kubelet wins the race and has deleted the pods. These don't cause test- to fail.
per offline discussion with Mike, I think he's right. I'll check the api server's code.
That's my expectation, too (that grace period being zero should bypass all graceful deletion code and just immediately delete the pod).
When the grace period is zero, api server update the storage(), and then delete the pod from the storage (). The check of if grace period is 0 is after the update(). It seems it should not do the update if grace periods is zero.
Do you agree that the above is a bug?
Hm, is the problem a missing or incorrect BeforeDelete handler?
I think we are dealing with two problems here: grace-period=0, the registry shouldn't do an update. client library should provide a Delete() method that takes an unique identifier, the "name+namespace" pair is ambiguous across time.
Thinking about this more, I wonder if the update is intentional. If so, we need some coordination so that both apiserver and kublet don't try to delete.
Grace period 0 is intentional, but may not be required for correct behavior. If the user signals delete immediately, we record that intent so it shows up in watch and can be a signal kubelet to terminate immediately (although we don't actually do that today, so we could walk back the behavior). We definitely need a precondition delete on uid.
There will never be a single owner of pod lifecycle because pods are bound to nodes, and so all components have to deal with races to delete.
I chose to err on the side of being explicit about user intent with the update to zero out termination. But technically explicit deletion can be the same signal.
A delete operation will also trigger a watch event, so kubelet will be signaled even if we skip the update right? [edit] Oh I see you answered right before I asked.
It should be, yes. In the interest of fixing this test in the short term, I think the easiest thing to do might be to turn off the node controller for this section of the test. (Is it needed by anything in the test?) But we should file at least two issues: "kubectl delete can race with graceful deletion"-- Things have to handle races, but there shouldn't be a race where you delete something and your delete fails because your own delete caused another component to issue a delete for the same thing... As far as I can see from the code, this could happen if apiserver context-switched at the wrong times (even if the more common thing of an old delete getting a new object is the issue at hand). controllers should delete via UID (or namespace/name + RV) -- I wouldn't be surprised if there's already an issue for this.
+1 to resourceVersion and uid preconditions on delete. Yes, maybe we shouldn't update the grace period before deletion in the case that the grace period is 0. Why can't we use unique pod names in the test? Turning off node controller seems like a bad idea. Index by uid is .
I don't see any flag option to turn of node controller. Pod's name is specified in the config file, is there any clean way to change the pod's name each time is called? Anyway, I'll try to make a PR to skip update if grace period is 0. That should alleviate this flake.
cc/ in case you are running into this issue.
As for how to change the name: we could use sed to replace the name. Or we could change the namespace.
This should be fixed by . I filed for deletion via UID or namespace/name+resourceVersion.
This happened again:
This error "] pods "redis-proxy" not found" here seems related to the error in .
I think the issue in is the same. The last few lines of logs: The symptom seems to align with my hypothesis in suggests the nodecontroller has successfully deleted a pod (probably "redis-master"), so kubectl fails with . I'll add more logs in nodecontroller to make sure this is the problem. In fact, we knew this race condition when investigating , but we discovered other race condition that are more likely to happen so we didn't solve this one.
Sent to show what pod nodecontroller has deleted.
Anywhere where we delete lists of pods (and other resources), including those filtered by label selector, we should ignore not-found errors. There are such loops in the controller reapers for cascading deletion. cc
makes safely ignore the NotFound error. forces to issue a "GET" before submitting "DELETE". The "GET" will effectively check if the user is trying to delete a resource that is never created, so if later the API server sends a NotFound error in response to the "DELETE" request, kubectl can safely ignore it.

Discussion:
if pendingGraceful { return e.finalizeDelete(obj, false) } <ins> var ignoreNotFound bool = false var lastExisting runtime.Object = nil </ins> if graceful { out := e.NewFunc() lastGraceful := int64(0)

return nil, errDeleteNow } lastGraceful = *options.GracePeriodSeconds <ins> lastExisting = existing </ins> return existing, nil }), )

if lastGraceful > 0 { return out, nil } <del> // fall through and delete immediately </del> <ins> // If we are here, the registry supports grace period mechanism and // we are intentionally delete gracelessly. In this case, we may // enter a race with other k8s components. If other component wins // the race, the object will not be found, and we should tolerate // the NotFound error. See // https://github.com/kubernetes/kubernetes/issues/19403 for // details. ignoreNotFound = true // exit the switch and delete immediately </ins> case errDeleteNow: // we've updated the object to have a zero grace period, or it's already at 0, so // we should fall through and truly delete the object.

// delete immediately, or no graceful deletion supported out := e.NewFunc() if err := e.Storage.Delete(ctx, key, out); err != nil { <ins> // Please refer to the place where we set ignoreNotFound for the reason // why we ignore the NotFound error . if storage.IsNotFound(err) && ignoreNotFound && lastExisting != nil { // The lastExisting object may not be the last state of the object // before its deletion, but it's the best approximation. return e.finalizeDelete(lastExisting, true) } </ins> return nil, etcderr.InterpretDeleteError(err, e.QualifiedResource, name) } return e.finalizeDelete(out, true)