Change:
Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: description hoisted from I was about to file this. I spent yesterday triaging it and have determined the root cause. The issue is that the can "miss" handling the deletion of a if things happen quickly enough. Here's the flow: creates rs sees new rs, starts working on creating pods deletes rs 's see the rs deletion and calls , which adds the namespace/name of the rs to the work queue recreates rs with the exact same name as before 's see the rs addition and calls , which adds the namespace/name of the rs to the work queue (again) 's sync handler processes the entry from the queue the rs was recreated with the same name, when calls to get the rs, it's there (it's the 2nd one) This is a timing issue. The doesn't check the rs's UID, and if the order of operations is "just right", the controller's sync handler won't "see" the deletion, so it never calls to reflect that the rs was deleted. /sig apps Original description follows: In I had to re-run the integration test 4 times to get it to work. Log outputs:
/kind flake /sig testing
/remove-kind failing-test
I was about to file this. I spent yesterday triaging it and have determined the root cause. The issue is that the can "miss" handling the deletion of a if things happen quickly enough. Here's the flow: creates rs sees new rs, starts working on creating pods deletes rs 's see the rs deletion and calls , which adds the namespace/name of the rs to the work queue recreates rs with the exact same name as before 's see the rs addition and calls , which adds the namespace/name of the rs to the work queue (again) 's sync handler processes the entry from the queue the rs was recreated with the same name, when calls to get the rs, it's there (it's the 2nd one) This is a timing issue. The doesn't check the rs's UID, and if the order of operations is "just right", the controller's sync handler won't "see" the deletion, so it never calls to reflect that the rs was deleted. /sig apps
For a bit more clarity: When the rs informer sees a new rs, it calls When the rs informer sees a deleted rs, it calls The net effect is Once a key (name) is in the queue, pops it off and then does The problem here is that there's not necessarily a guarantee that the order of operations is always like this: --pop --normal processing --pop --not found -delete expectations --pop --normal processing When the test fails, this is what happens: --pop --normal processing -(note the change here - no pop!) --pop --normal processing I confirmed this by adding some additional print statements to and . The is always called at the "right" time. But sometimes isn't scheduled fast enough, so the new rs is already visible by the time is executed.
/milestone v1.13 FYI Yeah this is definitely becoming more of a problem, It shows up on triage, but I can't point to when it started become more flaky:
/priority critical-urgent
do you think the controller should change to differentiate between two replicasets with the same name but different UIDs?
cc
The flaky test is actually testing kubectl, so an option is to split this into two issues. We can fix the test to avoid this problem, which would fix the flakiness for people working on other things. And then separately come up with a fix for the issue in replicaset. I created a PR that updates the test to avoid creating replicasets with the same name:
The replica set controller needs a fix. Thanks for the clear description, /assign
Alternatively, perhaps we could fix this by adding UID to the key used in the delta fifo?
so flakes are definitely down now that we're working around this (thanks but do we have an issue for the "proper" fix proposed?
/reopen the root issue is not fixed, and I hate to lose investigative comments
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
took a guess at the priority
/assign janetkuo
We should get a fix in for the next release
As per comment, moving this to 1.14 /milestone v1.14
(tracking update - in progress )
This is an important bug but the fix in still needs work Shifting milestones /milestone v1.15
Hi! We are starting the code freeze for 1.15 tomorrow EOD. Just checking in to see if this issue still planned for the 1.15 cycle?
I think is pretty close, I left a few comments.
/milestone v1.16
Hello! I'm part of the bug triage team for the 1.16 release cycle and considering this issue is tagged for 1.16, but not updated for a long time, I'd like to check its status. The code freeze is starting on August 29th (about 1.5 weeks from now), which means that there should be a PR ready (and merged) until then. Do we still target this issue to be fixed for 1.16? If not please re-tag the issue to the planned milestone.
This issue and the relevant PR are open for a long time now. Should we move this to the next milestone or should we remove this from the milestone entirely?
A change this low-level needs more soak time. Moving to 1.17 /milestone v1.17
I think fixing the handlers and handling expectations in them will fix this issue /assign
Bug triage for 1.17 here. This issue has been open for a significant amount of time and since it is tagged for milestone 1.17, we want to let you know that the 1.17 code freeze is coming in less than one month on Nov. 14th. Will this issue be resolved before then?
I hope it will, PR is there, just a matter of getting a review/tag.

Discussion:
# Pre-condition: no replica set exists kube::test::get_object_assert rs "{{range.items}}{{$id_field}}:{{end}}" '' # Command <del> kubectl create -f hack/testdata/frontend-replicaset.yaml "${kube_flags[@]}" </del> <ins> #TODO(mortent): Remove this workaround when ReplicaSet bug described in issue #69376 is fixed local replicaset_name="frontend-no-cascade" sed -r 's/^(s*)(names*:s*frontends*$)/1name: '"${replicaset_name}"'/' hack/testdata/frontend-replicaset.yaml | kubectl create "${kube_flags[@]}" -f - </ins> # wait for all 3 pods to be set up kube::test::wait_object_assert 'pods -l "tier=frontend"' "{{range.items}}{{$pod_container_name_field}}:{{end}}" 'php-redis:php-redis:php-redis:' kube::log::status "Deleting rs" <del> kubectl delete rs frontend "${kube_flags[@]}" --cascade=false </del> <ins> kubectl delete rs "${replicaset_name}" "${kube_flags[@]}" --cascade=false </ins> # Wait for the rs to be deleted. kube::test::wait_object_assert rs "{{range.items}}{{$id_field}}:{{end}}" '' # Post-condition: All 3 pods still remain from frontend replica set