Change:
TestGrid: /kind flake /sig api-machinery
checking there is a ginkgo panic, independently of this issue, is it normal that we panic on these cases? coming back to the issue, the error comes from I couldn't spend much time, but we need to understand why the ratelimiter errors
It started to flake more after 2023/10/10. Before , we use will not return context err. After , we use and it may fail for .
There is something odd here, why they will exceed the deadline , it is just the second try I noticed previous loops use a different timeout is it possible that we are using the wrong context here? if that is the case, why it cancel the request in flight?
Using assertions inside a callback as in is wrong. An error should be returned instead. Or better yet, don't use ... An assertion inside a is not ideal either, but I think would handle it better than .
The context is the one from That looks right to me. In , there is a "normal" , but given the lack of information in the log output it is impossible to tell how long it has been polling. seems normal to me, too, if is called shortly before the 5 min timeout. The then isn't the number of times that polled, it's the number of times that the rate limiter retried the .
mentioned that as well and This may finally be changed to something like below:
ok, so it really times out, there is a 5 mins gap, between the error and the previous log Can anybody get the test right per Patrick suggestions? asserting inside the loops and creating panics is definitely wrong
don't use these jobs to debug this problem, those create a multi master cluster and there is a loadbalancer in between that makes it hard to debug,
+ should work together to fix this issue. FYI:
Reporting the timeout will become better, but it doesn't help when the expected state really isn't reached.
Given that there is additional checking in it might be best to use a custom matcher that is specific to this particular test.
Another idea that I keep coming back to, but haven't turned into an issue or some actual code: if some failure occurs, then we need to know about the state of the cluster and events that may have been relevant for the test. was written to gather such information. I think it would be nicer if this was part of a report after a failure. That report then can also be more detailed than log messages that get written regardless of whether the test passes or not. One way to do that would be to gather events in that watcher in memory instead of logging them and when generating the failure message in the custom matcher, include that information. That would be limited to this test. Ideally, the framework should make that easier or even do it automatically - not sure yet how... perhaps by manipulating the Gingko failure handler while a framework instance is active. There's one potential downside of not dumping information during a successful run: it becomes impossible to compare a "good" run with a "bad" run. But that's already true today for the CI logs because we discard the log output from "good" runs. For running tests locally, a flag could enable dumping this information also for passing tests. It could even be on by default and only get turned off in the CI? Or perhaps always be enabled?
that is already dumped today, isn't it?
yeah, that's broken. it will panic on the first GET call from the RQ client if it fails. first thing to fix here is to remove all the assertions from the PollUntilContextTimeout handler function. there could be log messages and/or state can be stored, then once the poll is done state can be asserted outside of the handler.
Something is dumped. But I am not sure how complete it is. I think it also only dumps events that still exist. If some were garbage-collected, then they won't show up. Not sure whether that is a problem. Anyway, definitely needs to be looked into.
I don't know the owner of the resource quota controller, can someone check why the resource quota controller takes more than 5 minutes and fails the test?
/assign Having a look
From kube-controller-manager logs:
the flake is caused by a metrics-server addon deployment failure? RS controller fails to deploy metrics server pods because of a quota problem -the RQ controller's discovery client fails to get a fresh state for the metrics APIservice. i could be reading this wrong... there is also this:
is an example where this test is PASSING while metrics server discovery is failing. We should probably still figure out why the metrics server is not starting (cc but it MIGHT be a red herring here.
I filtered and then diffed the kube-controller-manager logs between a passing test and a failing test, looking for any difference in behavior in the RQ controller, I was not able to observe any difference in behavior from the logs..
/sig instrumentation /cc
/assign Found the race, tricky one, checking The test fail after modifying the status and waiting for the resourcequotacontroller to reconcile the spec to status again. However, the controller ignores updates to the status on purpose assuming is the only owner of the status and waits for the full reconcile loop to recover it If we observer this failure, the first check is at 03:30:44.387 and it times out at 03:35:44.388 if we check the kube-controller-manager logs we can see that the resourcequotacontroller loop runs just before and after we just need to take this into account on the test so the poll loop always get one of the full calculation usage The problem is that this period is random and it seems the defaul is 12 hours ???? on tests is 3 mins what is the value of this field for kubeadm? I answer myself, I just checked in a kind cluster and is the default
ok, my bad, I didn't follow the code right, the resync period for the resource quota controller is an option in and the default is 5 minutes that matches with the observation on the failed test , everything adds up now
/triage accepted

Discussion:
}) framework.ExpectNoError(err, "failed to locate ResourceQuota %q in namespace %q", patchedResourceQuota.Name, ns) <del> err = wait.PollUntilContextTimeout(ctx, 5*time.Second, 5*time.Minute, true, func(ctx context.Context) (bool, error) { </del> <ins> // the resource_quota_controller ignores changes to the status so we have to wait for a full resync of the controller // to reconcile the status again, this full resync is set every 5 minutes by default so we need to poll at least one // minute more just in case we we start to poll just after the full resync has happened and he have to wait until // next full resync. // Ref: https://issues.k8s.io/121911 err = wait.PollUntilContextTimeout(ctx, 5*time.Second, 6*time.Minute, true, func(ctx context.Context) (bool, error) { </ins> resourceQuotaResult, err := rqClient.Get(ctx, rqName, metav1.GetOptions{}) <del> framework.ExpectNoError(err) </del> <ins> if err != nil { return false, nil } </ins> <del> if apiequality.Semantic.DeepEqual(resourceQuotaResult.Spec.Hard.Cpu(), resourceQuotaResult.Status.Hard.Cpu()) { gomega.Expect(*resourceQuotaResult.Status.Hard.Cpu()).To(gomega.Equal(resource.MustParse("1")), "Hard cpu value for ResourceQuota %q is %s not 1.", repatchedResourceQuota.Name, repatchedResourceQuota.Status.Hard.Cpu().String()) gomega.Expect(*resourceQuotaResult.Status.Hard.Memory()).To(gomega.Equal(resource.MustParse("1Gi")), "Hard memory value for ResourceQuota %q is %s not 1Gi.", repatchedResourceQuota.Name, repatchedResourceQuota.Status.Hard.Memory().String()) </del> <ins> if *resourceQuotaResult.Spec.Hard.Cpu() == *resourceQuotaResult.Status.Hard.Cpu() { if *resourceQuotaResult.Status.Hard.Cpu() != resource.MustParse("1") { framework.Logf("Hard cpu status value for ResourceQuota %q is %s not 1.", repatchedResourceQuota.Name, resourceQuotaResult.Status.Hard.Cpu().String()) return false, nil } if *resourceQuotaResult.Status.Hard.Memory() != resource.MustParse("1Gi") { framework.Logf("Hard memory status value for ResourceQuota %q is %s not 1Gi.", repatchedResourceQuota.Name, resourceQuotaResult.Status.Hard.Memory().String()) return false, nil } </ins> framework.Logf("ResourceQuota %q Spec was unchanged and /status reset", resourceQuotaResult.Name) return true, nil } <ins> framework.Logf("ResourceQuota %q Spec and Status does not match: %#v", resourceQuotaResult.Name, resourceQuotaResult) </ins> return false, nil }) <del> framework.ExpectNoError(err) </del> <ins> if err != nil { framework.Failf("Error waiting for ResourceQuota %q to reset its Status: %v", patchedResourceQuota.Name, err) } </ins> }) })