Change:
Which jobs are flaking: pull-kubernetes-e2e-gce Which test(s) are flaking: [sig-api-machinery] Garbage collector should not be blocked by dependency circle [Conformance] [sig-api-machinery] Garbage collector should orphan RS created by deployment when deleteOptions.PropagationPolicy is Orphan [Conformance] [sig-api-machinery] Garbage collector should support cascading deletion of custom resources [sig-api-machinery] Garbage collector should keep the rc around until all its pods are deleted if the deleteOptions says so [Conformance] [sig-api-machinery] Garbage collector should delete RS created by deployment when not orphaning [Conformance] [sig-api-machinery] Garbage collector should delete pods created by rc when not orphaning [Conformance] [sig-api-machinery] Garbage collector should delete jobs and pods created by cronjob Testgrid link: Reason for failure: timeout waiting for deletion to complete Anything else we need to know: seems to have gotten much worse on/after 1/24/2020: See the time period on 2/11 for lots of examples number 2 flake according to /sig api-machinery
from the failure in gc events related to the test: test timed out: gc events related to the test:
cc
/assign thank you for fixing it!
top flake according to
looking at the debug logs from the last several failures, filtering remaining pods to name+finalizers+ownerReferences with: jq '[.[] | {"metadata":{"name","finalizers":(.metadata.finalizers // []), "ownerReferences": .metadata.ownerReferences}}]'
no consistent pattern between failures, GC just seems to have been running slower and sometimes timed out the test. tracked likely performance degradation to , fix in
note that should return performance to pre-1/24 levels, but this test was flaky even before that - *circle
Further debugging of GC lag issues in kube-controller-manager in e2e (after merged) Combined log of e2e+kcm+apiserver for the timeout test failure from is at The RC is deleted by the e2e test: GC starts running attemptToDelete on the rc and all pods within a second: attemptToDeleteItem on the RC (started at 17:37:28.) takes ~17 seconds to process (end result is calling processDeletingDependentsItem on all the child pods): attemptToDelete on the pods (started at 17:37:28. - 17:37:28.) also takes ~17 seconds to complete (end result is GC making a DELETE API call on each pod): 2 seconds of ~17 second delay is the GC controller rebuilding its restmapper via discovery: The other 15 seconds are unaccounted for. Then, GC reacts to changing API group availability by performing its periodic resync of informers, which fails once (because another parallel e2e test removed a CRD at the same time), retries, takes a total of 39 seconds, and blocks item processing until complete: After resyncing monitors, it immediately adds all the items that received updates or were deleted while it was resyncing, including the 10 deleted pods (which no longer exist) and their parent RC: This time there is a ~16 seconds delay from attemptToDelete starting to handle the RC (at 17:38:26.) and the foreground deletion finalizer being removed: But the e2e test has already failed:
cc
The informer resync is not unexpected, because we have lots of parallel e2e tests that add/remove CRDs. The extra time for GC to recover from an API group sync failure (39 seconds, in this case) is not included in most of the GC e2e timeout periods. The 17 and 15 second execution times for attemptToDelete are worthy of more investigation. Combined, they made GC easily exceed the 60 second period (actual time: the test waits for the RC to be removed
adding fine-grained trace.LogIfLong statements to attemptToDelete in particular might be helpful in revealing exactly where we are spending so much time
Tried this out with The long delays outside of sync.Monitors are almost entirely due to the latency of I'm guessing that is fetching from discovery in this codepath for the same reason does: another parallel e2e test removed a CRD at the same time. more details:
Traced this down to the discovery 's cache refresh. See for a fix.

Discussion:
return err } <ins> wg := &sync.WaitGroup{} resultLock := &sync.Mutex{} </ins> rl := map[string]*cacheEntry{} for _, g := range gl.Groups { for _, v := range g.Versions { <del> r, err := d.serverResourcesForGroupVersion(v.GroupVersion) rl[v.GroupVersion] = &cacheEntry{r, err} if err != nil { utilruntime.HandleError(fmt.Errorf("couldn't get resource list for %v: %v", v.GroupVersion, err)) } </del> <ins> gv := v.GroupVersion wg.Add(1) go func() { defer wg.Done() defer utilruntime.HandleCrash() r, err := d.serverResourcesForGroupVersion(gv) if err != nil { utilruntime.HandleError(fmt.Errorf("couldn't get resource list for %v: %v", gv, err)) } resultLock.Lock() defer resultLock.Unlock() rl[gv] = &cacheEntry{r, err} }() </ins> } } <ins> wg.Wait() </ins> d.groupToServerResources, d.groupList = rl, gl d.cacheValid = true

func TestPartialPermanentFailure(t *testing.T) { fake := &fakeDiscovery{ groupList: &metav1.APIGroupList{ <del> Groups: []metav1.APIGroup{{ Name: "astronomy", Versions: []metav1.GroupVersionForDiscovery{{ GroupVersion: "astronomy/v8beta1", Version:      "v8beta1", }, { GroupVersion: "astronomy2/v8beta1", Version:      "v8beta1", }}, }}, </del> <ins> Groups: []metav1.APIGroup{ { Name: "astronomy", Versions: []metav1.GroupVersionForDiscovery{{ GroupVersion: "astronomy/v8beta1", Version:      "v8beta1", }}, }, { Name: "astronomy2", Versions: []metav1.GroupVersionForDiscovery{{ GroupVersion: "astronomy2/v8beta1", Version:      "v8beta1", }}, }, }, </ins> }, resourceMap: map[string]*resourceMapEntry{ "astronomy/v8beta1": {

func TestPartialRetryableFailure(t *testing.T) { fake := &fakeDiscovery{ groupList: &metav1.APIGroupList{ <del> Groups: []metav1.APIGroup{{ Name: "astronomy", Versions: []metav1.GroupVersionForDiscovery{{ GroupVersion: "astronomy/v8beta1", Version:      "v8beta1", }, { GroupVersion: "astronomy2/v8beta1", Version:      "v8beta1", }}, }}, </del> <ins> Groups: []metav1.APIGroup{ { Name: "astronomy", Versions: []metav1.GroupVersionForDiscovery{{ GroupVersion: "astronomy/v8beta1", Version:      "v8beta1", }}, }, { Name: "astronomy2", Versions: []metav1.GroupVersionForDiscovery{{ GroupVersion: "astronomy2/v8beta1", Version:      "v8beta1", }}, }, }, </ins> }, resourceMap: map[string]*resourceMapEntry{ "astronomy/v8beta1": {