Change:
pull-kubernetes-unit TestSyncPastDeadlineJobFinished June 14th Probably after /sig apps
/priority critical-urgent /assign /triage accepted
I'm going to mitigate this by increasing the timeout. However, we should refactor the test (and job controller) to be able to use a fake clock that we can use to make the test more reliable.
I have been working on getting this refactoring done as per the comment from previous PR's review comments. I will have a draft PR out for review by tomorrow.
Sounds good. Please include this PR in the cherry-picks
I see the cherry picks are still open. I will pull your change to increase the timeout into them rightaway .
As part of the new changes. I am attaching a to the Job and using that for all time related operations. In the UT, I am replacing that with a fake clock. I am also replacing the with a custom based type for UT that is backed by the fake clock. Is this the right approach or is there a different design you have in mind ?
That sounds about right, thanks
/assign
I was looking through the links and this one caught my eyes. Something to be concerned about ? I don't think this is anything to do with the test cases we that are causing the flake but I am not entirely sure. Any suggestion ?
No, that is a setting for the test suite. It's expected.
Looks like the 10s bump up you did is not good enough it looks like. I still see flakes in that test that is ongoing. Please take a look at the refactoring PR done to address this behavior when time permits.
The fact that flakiness didn't improve whatsoever made me realize that we missed something Thanks for your PR, I'll check it soon, but we probably don't want to cherry-pick that.
oh no! :( I should have been a bit more careful with that one. Until you mentioned it, I did not even realize that.
It happens. Please include this PR in the cherry-picks instead :)
Sure.
Done.
Awesome, let's get the PR merged and let it soak in the CI. Then we can ping back the release-managers to approve the cherry-picks
/close The test looks stable now
Closing this issue. <detailsIn response to : /close The test looks stable now Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
It seems to be flaky these days. Have we got any related changes around here? <img width="1112" alt=" 2022-12-23 19 10 01" src="https://user-"/reopen
Reopened this issue. <detailsIn response to : It seems to be flaky these days. Have we got any related changes around here? Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
There was a recent refactoring change that was merged Checking to see if that is the reason behind the new flakes.
Definitely seem to be a side effect of the changes merged. I am seeing flaky results for both cases even with fake clock attached to the controller where timing is controlled. I am trying to narrow down the reason for the flakiness. Will update the thread shortly. cc
Think I narrowed it down to an odd behavior I am noticing with the fake clocks and the re-queue. Would it be safer to just replace the test section of the call to perform the directly ?
Able to reproduce the issue by running in a loop. over a series of retries. Happened to fail 4 times while running the following
cc
I am thinking of turning the test to look something like this.
Theories (could be a combination of both): Somewhere in the controller code we forgot to replace a real clock for the clock passed through the arguments this line sets the time too old: Now is already one second later, so the controller adds the condition.
I already Verified this. This is not the case. No references to directly anywhere on the other than duration parameters. Yes. This was exactly the reason why couple of tests were flaking with the error..
I created an alternate PR
/reopen It remains flaky as discussed here:
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
here is what I had noticed while checking this earlier and discussing with If you see the above, the start Time gets set once when invoked without the start time being set already. The next assertion to check if the value was set or not passes just fine. When you invoke the sync job again, where it fetches the item to process the queue again, it seem to have not updated the job that has the start time. So it goes and sets the start time again, but this time it does so to the current value. So, the more often this happen, the longer the poll loop will run. Without the the clock would never advance so it would forever be setting the same value back and forth and condition would never be satisfied. The current helps with that behavior better because each time you are in the wait loop, you are advancing the clock. But that still doesn't completely take the original issue away. The main issue is that the lister used to fetch the item to process the queue again in sync job doesn't contain the updated entity, so it is basically a game of roulette to see if the clock actually gets set and update the right time for the test to pass. I see that included an extra set of code to put the job under indexer, which I thought could have mitigated the issue, because getting back to the queue already has the set to a value. Would this work ? Since we are using the fake clock for the workqueue as well, the item will only be to the queue if the clock is advanced first is it not? (Sorry, If This is a silly question and I misunderstood what you mentioned above) Debugging further, I noticed something else too Especially the following part.
The workqueue is actually two queues: the first one holds the items until is fulfilled. The second one is just a FIFO queue. We need to check whether the first queue has elements.

Discussion:
manager, sharedInformerFactory := newControllerFromClientWithClock(clientset, controller.NoResyncPeriodFunc, fakeClock) manager.podStoreSynced = alwaysReady manager.jobStoreSynced = alwaysReady <ins> manager.expectations = FakeJobExpectations{ controller.NewControllerExpectations(), true, func() { }, } </ins> ctx, cancel := context.WithCancel(context.Background()) defer cancel() sharedInformerFactory.Start(ctx.Done())

if err := sharedInformerFactory.Batch().V1().Jobs().Informer().GetIndexer().Add(job); err != nil { t.Fatalf("Failed to insert job in index: %v", err) } <del> // This is needed because the fake clientset doesn't report created pods // to the informer, leading to unsatisfied expectations in syncJob that prevent the controller from setting the final condition. podIndexer := sharedInformerFactory.Core().V1().Pods().Informer().GetIndexer() setPodsStatuses(podIndexer, job, 0, 1, 0, 0, 0) </del> var j *batch.Job err = wait.PollImmediate(200*time.Microsecond, 3*time.Second, func() (done bool, err error) {

if err != nil { t.Errorf("Job failed to ensure that start time was set: %v", err) } <del> for _, c := range j.Status.Conditions { if c.Reason == "DeadlineExceeded" { t.Errorf("Job contains DeadlineExceeded condition earlier than expected") break } } </del> // Make sure the start time is in the informer cache. if err := sharedInformerFactory.Batch().V1().Jobs().Informer().GetIndexer().Add(j); err != nil { t.Fatalf("Failed to update job in cache: %v", err) } <del> manager.clock.Sleep(time.Second) err = wait.Poll(200*time.Millisecond, 3*time.Second, func() (done bool, err error) { </del> <ins> err = wait.Poll(100*time.Millisecond, 3*time.Second, func() (done bool, err error) { </ins> j, err = clientset.BatchV1().Jobs(metav1.NamespaceDefault).Get(ctx, job.GetName(), metav1.GetOptions{}) if err != nil { return false, nil } <del> if len(j.Status.Conditions) == 1 && j.Status.Conditions[0].Reason == "DeadlineExceeded" { </del> <ins> if getCondition(j, batch.JobFailed, v1.ConditionTrue, "DeadlineExceeded") { if manager.clock.Since(j.Status.StartTime.Time) < time.Duration(*j.Spec.ActiveDeadlineSeconds)*time.Second { return true, errors.New("Job contains DeadlineExceeded condition earlier than expected") } </ins> return true, nil } <ins> manager.clock.Sleep(100 * time.Millisecond) </ins> return false, nil }) if err != nil {