Change:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: I did a review of and found out a case where a SharedIndexInformer can miss a notification. Follows an example scenario where this could happen. 1) A SharedIndexInformer is running, and its reflector is WATCHing. 2) A delete notification for an API object X with namespaced name N1 and UID U1 which was in the informer's cache is received. 3) The reflector on X; the relevant store is a DeltaFIFO and its Delete method is . 4) The DeltaFIFO's Delete method invokes , which . Notice that the store of the reflector, i.e., the DeltaFIFO, is NOT the SharedIndexInformer's store/cache: . Assume that BEFORE X's deletion is popped from the DeltaFIFO and applied to the informer's cache, the following sequence of events take place: 5) The WATCH is interrupted because of a network partition. 6) While the reflector is not watching, a new API object Y with namespaced name N1 (same as X) and UID U2 is created. 7) The partition disappears and the reflector does a relist against the API server; there's a "hole" between the last watch notification and the list results; assume Y is in the LIST results. 8) The reflector will load the LIST results in the store/DeltaFIFO with which in turn calls the DeltaFIFO's method. The DeltaFIFO's replace method attempts to add to the DeltaFIFO a delta of type for every object in the LIST results: when processing Y it will which should . But won't enqueue Y's delta, because it enqueues deltas of type only if there's not already a delta of type for the given object ID as the last delta for that object ID (see and ), to avoid recreating a deleted object during a periodic resync triggered by the SharedIndexInformer resync period. Object IDs are namespaced names, and remember that the last delta for Y's namespaced name in the DeltaFIFO is X's deletion. The consequence is that Y's creation delta is not to the DeltaFIFO: Y is not to the SharedIndexInformer's cache and its creation notification handler does not execute. The SharedIndexInformer missed Y's creation. What you expected to happen: I expected SharedIndexInformers to get all notifications from the API server. Anything else we need to know? Missed notifications should be rare: for them to occur a delta must stay in the DeltaFIFO without being processed for long enough for a new List against the API server to be completed. Also, touches the reflector code that handles list and watches, and it might introduce changes relevant to this bug. Proposed solution: There's a straightforward fix: get rid of the check that skips enqueuing of a delta for object ID X if the last delta to the DeltaFIFO (and still in it) for X is of type ; is the code to remove. Notice that such check applies to deltas of type only. deltas are triggered only by relists and periodic resyncs. In the case of relists the check is harmful as it's the cause of this bug. In the case of periodic resyncs, that check is useless because it will either evaluate to or not execute; in fact, during a period resync, and this condition is checked before the check that would be removed, OTOH, if this condition is true and processing goes on the check that would be removed can only evaluate to (hence the delta for X is enqueued). If this solution has consensus I'd be happy to open a PR.
/assign
/sig api-machinery
cc
cc
xref ,
cc
/cc
This looks similar to failures we were seeing in kube-proxy while doing a rolling master upgrade. It was repeatable enough that someone hacked a workaround that I later spotted - this is close enough to the description that I’m suspicious that it was real /priority critical-urgent Since any missed informer event can cause all sorts of safety violations.
Note that rolling master upgrades break watches en masse, so a backlogged informer (contention / other locking / busy machine) would be more likely so see it the more caches it has.
A great first step would be to craft a test that demonstrates the issue repeatably, to help verify potential fixes resolve it.
IYO what would be the most appropriate kind of test, a unit test for the DeltaFIFO or something higher-level like integration/e2e tests?
The lower-level the test will be the better. So ideally unit test.
the lowest level that demonstrates the problematic behavior. if the issue is with the interaction between reflector and shared informer, then ideally something at the shared informer level
When the reflector calls it is expected that for each object O with ID X in a delta is to the DeltaFIFO, but that is (erroneously) not the case if the DeltaFIFO already stores a delta for ID X . So it's the DeltaFIFO which causes troubles and should be tested. But that alone does not show the user-visible problem; for a test to fail because a SharedIndexInformer missed a notification, the test would have to be at the SharedIndexInformer level.
That seems reasonable. I'd expect a DeltaFIFO test that sets up the described scenario and exercises both the Relist/Replace and Resync cases described here:
Sounds good, I'm at work.
is the testcase literally just adding this to But for some reason this hangs in the Pop(f).
Yes, the test looks very similar to what you posted, but there's a mistake: the expected result must be a single with the two that in your example are the only elements of the two 1-element Deltas that make up , respectively.
(that would be caught and fail the test immediately by changing to )
Aha, you're totally right :) Had to run off to lunch before spending more time on it.
DeltaFIFO only de-dupes deletions, but your example would do effectively extend that to compressing Delete+Sync=Sync. You used to be able to build smarter compression off DeltaFIFO (eg ) but nobody actually used the DeltaCompressor stuff so we removed it. And nobody above DeltaFIFO expects compressed events now :( In your example I'd be a bit worried about the implications of not deleting the object, since the Sync-ed one probably has a different UUID and thus technically not be the same object as the deleted one. Might not be a big issue, but this is critical path code so I'd personally rather be a bit conservative.
I agree about not compressing. It's a little hard to reason about what this sync event should be when we know that relists can currently time-travel backwards. Edit: it looks like shared informer code will if it follows a delete event.
I was this morning in the "sig-api-machinery" channel of the "Kubernetes" Slack team. My reading of the code had me expecting that I'd get a delete followed by an add.
This issue is specifically focused on how a queued delete event (typically via directly observing a delete watch event or via absence in a relist/replace) followed by an observed add (via presence in a relist) is handled by DeltaFIFO. your question is slightly broader/different, and has multiple facets: UID changes considered when handling reflector object add/delete/update events? They are not; if an object is deleted and recreated between the reflector dropping a watch and relisting, the reflector will enqueue an Sync event with the object with the new uid (which the shared informer will turn into an Update event) delete/add events is actively compressed into an update event? I don't think so, but it's been a while since I've traipsed through reflector/fifo code, so I'm not positive (and since the Update with a new uid can occur from scenario 1, informer clients have to be prepared for uid to change on update anyway)
Listeners need to handle Sync too. And the only two in-tree users of NewDeltaFIFO are and and they look like they correctly handle Sync.
Edit: yep you're right, I was looking at slightly older code.
if this happens, I think it should happen on top of the DeltaFIFO, not in the DF. Let's keep DF simpler since it's (a) complicated enough and (b) critical path code. probably the right place for compression, if any happens. IMHO it's correct that Deletes don't get compressed, eg that Delete + Add is not compressed into Update or something like that.
I agree, though that specific example can happen naturally if it occurs in a window while the reflector isn't getting watch events.
Just to make sure I understand this properly, does this scenario involve an object in the cache with name "x" and UID "u1," then the reflector loses its watch, reestablishes it and lists all the relevant objects in the cluster, finds one named "x" with UID "u2," and since it already has an "x" in its cache, translates this discovery into a call?
exactly (both namespace and name are part of the cache key, uid is not)
does this mean we couldn't use the argument to DeltaFIFO's Replace() to work around whatever is talking about (if that problem still exists)?
clients can only use resourceVersion to check equality, not older-than/newer-than
As noted in Slack, there is now careful commentary in about what is promised. There is no promise that a delete followed by an add is delivered as two notifications; it is allowed to be delivered as one update notification.

Discussion:
return b } <del> // willObjectBeDeletedLocked returns true only if the last delta for the // given object is Delete. Caller must lock first. func (f *DeltaFIFO) willObjectBeDeletedLocked(id string) bool { deltas := f.items[id] return len(deltas) > 0 && deltas[len(deltas)-1].Type == Deleted } </del> // queueActionLocked appends to the delta list for the object. // Caller must lock first. func (f *DeltaFIFO) queueActionLocked(actionType DeltaType, obj interface{}) error {

return KeyError{obj, err} } <del> // If object is supposed to be deleted (last event is Deleted), // then we should ignore Sync events, because it would result in // recreation of this object. if actionType == Sync && f.willObjectBeDeletedLocked(id) { return nil } </del> newDeltas := append(f.items[id], Delta{actionType, obj}) newDeltas = dedupDeltas(newDeltas)

} } <ins> // TestDeltaFIFO_replaceWithDeleteDeltaIn tests that a `Sync` delta for an // object `O` with ID `X` is added when .Replace is called and `O` is among the // replacement objects even if the DeltaFIFO already stores in terminal position // a delta of type `Delete` for ID `X`. Not adding the `Sync` delta causes // SharedIndexInformers to miss `O`'s create notification, see https://github.com/kubernetes/kubernetes/issues/83810 // for more details. func TestDeltaFIFO_replaceWithDeleteDeltaIn(t *testing.T) { oldObj := mkFifoObj("foo", 1) newObj := mkFifoObj("foo", 2) f := NewDeltaFIFO(testFifoObjectKeyFunc, keyLookupFunc(func() []testFifoObject { return []testFifoObject{oldObj} })) f.Delete(oldObj) f.Replace([]interface{}{newObj}, "") actualDeltas := Pop(f) expectedDeltas := Deltas{ Delta{Type: Deleted, Object: oldObj}, Delta{Type: Sync, Object: newObj}, } if !reflect.DeepEqual(expectedDeltas, actualDeltas) { t.Errorf("expected %#v, got %#v", expectedDeltas, actualDeltas) } } </ins> func TestDeltaFIFO_requeueOnPop(t *testing.T) { f := NewDeltaFIFO(testFifoObjectKeyFunc, nil)