Change:
Running latest from master (currently commit ), I'm seeing TestMonitorNodeStatusEvictPodsWithDisruption is flakey: test -c ./pkg/controller/node true; do if ! ; then break; fi; done for failure I've narrowed it down , which is supposed to process all the items (nodes) in the queue. This queue has a capacity of 1, and at least in the test code, a refill rate of QPS. If the call to fails , it will stop processing wherever it happened to be in the queue. I've some extra logging locally, and when it fails, I see it processing nodes 1 and 2 before TryAccept fails. The issue is that the pod it's trying to evict is on node0, and it doesn't process node0 before TryAccept returns false. cc
I'm surprised we haven't hit this in jenkins yet
and removed the "leak" option referred to in the godoc, but it seems like it kept the leak behavior, not the retry behavior when the rate limiter is hit
Still seeing this issue
I didn't understand how this is related to sig/scheduling. Am I missing anything here?
guessing the flake is related to this line: anything on the order of milliseconds is going to be unreliable in testenvs that sometimes peg the CPU with parallel tests or it's related to the single item burst and the "abort when TryAccept fails" behavior
I can extend this sleep to something bigger, but I don't have cycles to debug it properly:/
any thoughts on that?
Generally we retry when TryAccept fails, but not in tests. So either we need to rewrite the test, or lengthen the sleep time. It's not perfect, but there's no way for "wait until rate limiter is reloaded".
I was working on this issue and I tried the change that you suggested and I notice that it fails with this change too() and I think rewriting the test with retry seems to be the option as I tried increasing 5000 ms and still it failed. I am new to the codebase, if there is piece of code where retry is available, I can check on that.
- what you wrote is concerning. I wish I had time to debug it:/ Frankly - those tests are old. Like really, really old. So they start to deserve a rewrite. On the flip side - we'll need to rewrite them when making taint-based evictions default, so we'll need to allocate time for that anyway. If lengthening sleep time reduce the flakyness I vote for doing that, and just wait for upcoming rewrite.
having a delay in the loop before caused it to run for close 4 hours without terminating on 8GB,4 core machine. Is this something normal? I know, is not happy with this change but since we are rewriting this test anyways, this could be a work around that removes the flakyness. Let me know your thoughts. If this does work, can we remove the flake label and keep this issues as this is not a release blocker for 1.6?
- I don't understand which delay in which loop you mean. You a delay inside the "Try" call? What do you mean by 'run for close 4 hours without terminating'? That the test didn't fail? If so, then I'm all for such solution for stopping the bleeding.
, Yes I introduced a sleep in loop for zone, like Yes the test didn't fail but it didn't complete either even after 4 hours, so I am wondering if this causes timeout in testing infra.
It's strange indeed... Do you have any logs for the test by any chance? Adding this delay shouldn't be noticeable, as it just inserts some delay here: Sleeping under the lock is usually not the best idea, but 50 microseconds in test shouldn't make a lot of difference...
Unfortunately no, I can try reproducing this today. Probably by afternoon EST, I can have something, The way I am doing as of now is the way suggested by in the comments above. I can redirect that to a file and see. Let me know, if you are ok with approach.
I'm OK with anything that produces logs:) I generally do:
- out of curiosity. How often does this flake?
If I run my loop, it takes about 8-10 seconds to flake. , Marek Grabowski <:
I think I made a mistake. I did not take that while true loop out because of which it ran in infinite loop. I am sorry about that. It means that with that delay introduced that I mentioned earlier it never failed even after running for 4 hours, which means we got rid of atleast test flakyness. if you can confirm if it is working, I will raise a PR which introduces this delay.
giving the ratelimiter time to refill inside the loop is consistent with the theory that is causing problems. If the rate limiter runs out of tokens in Try(), it returns with a comment, but the unit test does not try again later. I'm having a hard time following the code to know if non-test invocations have the same problem or if this is just a unit test issue.
The sleep addition seems to have helped. rgudimet <:
It's test only problem: Runs in , so it'll retry in the next round. I guess the problem here was actually introduces later than what suggests - I don't think that the removal of leaky behavior was problematic, but introducing a test that evicts more than one Node was. This means that if processing queue is fast enough, then rate limiter fails to reload, so adding the sleep there is actually a correct solution (as long as we don't have good rate limiter mock/testable rate limiter).
Thanks for confirmation. I will raise a PR for that now. As far as I understood, the issue is with unit test. If rate limiter fails to reload for a particular node, it cannot retry(unless the we separate out the logic which tells if the rate limiter has reloaded or not. - if we have that logic we can ask Try to run again by returning a false just like ).
Still an issue: https://k8s-
but not a blocker (the test is at fault)
Thanks, I will check this out after 1.6 release hopefully, with rewrites..
Has there been any recent incidents of this issue? It's marked for 1.7.
Closing - shows near nothing.
- this flakes out sporadically, I was able to reproduce this when debugging on my system, if I run it for long time.
https://k8s-

Discussion:
if err := nodeController.monitorNodeStatus(); err != nil { t.Errorf("%v: unexpected error: %v", item.description, err) } <del> // Give some time for rate-limiter to reload time.Sleep(500 * time.Millisecond) </del> for zone, state := range item.expectedFollowingStates { if state != nodeController.zoneStates[zone] { t.Errorf("%v: Unexpected zone state: %v: %v instead %v", item.description, zone, nodeController.zoneStates[zone], state) } } <del> zones := testutil.GetZones(fakeNodeHandler) for _, zone := range zones { // Time for rate-limiter reloading per node. time.Sleep(50 * time.Millisecond) nodeController.zonePodEvictor[zone].Try(func(value TimedValue) (bool, time.Duration) { uid, _ := value.UID.(string) deletePods(fakeNodeHandler, nodeController.recorder, value.Value, uid, nodeController.daemonSetStore) return true, 0 }) } podEvicted := false for _, action := range fakeNodeHandler.Actions() { if action.GetVerb() == "delete" && action.GetResource().Resource == "pods" { podEvicted = true </del> <ins> var podEvicted bool start := time.Now() // Infinite loop, used for retrying in case ratelimiter fails to reload for Try function. // this breaks when we have the status that we need for test case or when we don't see the // intended result after 1 minute. for { podEvicted = nodeController.doEviction(fakeNodeHandler) if podEvicted == item.expectedEvictPods || time.Since(start) > 1*time.Minute { </ins> break } }

} } <ins> // doEviction does the fake eviction and returns the status of eviction operation. func (nc *nodeController) doEviction(fakeNodeHandler *testutil.FakeNodeHandler) bool { var podEvicted bool zones := testutil.GetZones(fakeNodeHandler) for _, zone := range zones { nc.zonePodEvictor[zone].Try(func(value TimedValue) (bool, time.Duration) { uid, _ := value.UID.(string) deletePods(fakeNodeHandler, nc.recorder, value.Value, uid, nc.daemonSetStore) return true, 0 }) } for _, action := range fakeNodeHandler.Actions() { if action.GetVerb() == "delete" && action.GetResource().Resource == "pods" { podEvicted = true return podEvicted } } return podEvicted } </ins> // TestCloudProviderNoRateLimit tests that monitorNodes() immediately deletes // pods and the node when kubelet has not reported, and the cloudprovider says // the node is gone.