Change:
<!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): No What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): , , Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Environment: Cloud provider or hardware configuration: bare metal OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: , deployed with Others: What happened: I am using a scheduledjob with . An instance of my job has taken a long time to finish, thus skipping many start times. Ever since, the scheduledjob will not schedule anything new. Controller manager log has: What you expected to happen: I expect the scheduledjob not to be stuck not scheduling anything anymore. With , I expect that it should just re-schedule something as soon as the previously running job has completed. The "Too many missed start times to list" safeguard appears to apply mostly to cases where job concurrency is allowed to avoid creating a lot of jobs in case of timestamp issues. ` How to reproduce it (as minimally and precisely as possible): The issue shows up after about 100 minutes of this running. Anything else do we need to know:
cc
sorry for the long delay in response, there were a few issues solved in later versions of ScheduledJob, can you verify this against latest 1.4.x? I've tried cherry-picking all of the crucial ones, like: and I'll try to verify this on my own as well.
I am still getting the same issue with
thanks for the info, will dig deeper into this one.
Thanks a lot any chance a potential fix could make it in 1.5?
if possible - yes, I'll try to backport it to 1.5
did you look into this? If not, let me know if you want me to pick it up and work on a PR
yeah, go ahead and ping me in the PR.
I just thought of something which might make sense. We've modified the error message to make it more "actionable", and the user can set a deadline to avoid running into this. However, the error message is still only controller level AFAIK and thus not exposed to the user. From the user's perspective, there is not way to tell why the CronJob is not scheduling the next instance. For these reasons, maybe we should also emit an event in the corresponding CronJob? I can send a small PR for this modification.

Discussion:
return T1 } <ins> func weekAfterTheHour() time.Time { T1, err := time.Parse(time.RFC3339, "2016-05-26T10:00:00Z") if err != nil { panic("test setup error") } return T1 } </ins> func justBeforeThePriorHour() time.Time { T1, err := time.Parse(time.RFC3339, "2016-05-19T08:59:00Z") if err != nil {

} var ( <del> shortDead int64                   = 10 longDead  int64                   = 1000000 noDead    int64                   = -12345 A         batch.ConcurrencyPolicy = batch.AllowConcurrent f         batch.ConcurrencyPolicy = batch.ForbidConcurrent R         batch.ConcurrencyPolicy = batch.ReplaceConcurrent T         bool                    = true F         bool                    = false </del> <ins> shortDead  int64                   = 10 mediumDead int64                   = 2 * 60 * 60 longDead   int64                   = 1000000 noDead     int64                   = -12345 A          batch.ConcurrencyPolicy = batch.AllowConcurrent f          batch.ConcurrencyPolicy = batch.ForbidConcurrent R          batch.ConcurrencyPolicy = batch.ReplaceConcurrent T          bool                    = true F          bool                    = false </ins> ) func TestSyncOne_RunOrNot(t *testing.T) { <ins> // Check expectations on deadline parameters if shortDead/60/60 >= 1 { t.Errorf("shortDead should be less than one hour") } if mediumDead/60/60 < 1 || mediumDead/60/60 >= 24 { t.Errorf("mediumDead should be between one hour and one day") } if longDead/60/60/24 < 10 { t.Errorf("longDead should be at least ten days") } </ins> testCases := map[string]struct { // sj spec concurrencyPolicy batch.ConcurrencyPolicy

"still active, is time, suspended":         {A, T, onTheHour, noDead, T, T, justAfterTheHour(), F, F, 1}, "still active, is time, past deadline":     {A, F, onTheHour, shortDead, T, T, justAfterTheHour(), F, F, 1}, "still active, is time, not past deadline": {A, F, onTheHour, longDead, T, T, justAfterTheHour(), T, F, 2}, <ins> // Controller should fail to schedule these, as there are too many missed starting times // and either no deadline or a too long deadline. "prev ran but done, long overdue, not past deadline, A": {A, F, onTheHour, longDead, T, F, weekAfterTheHour(), F, F, 0}, "prev ran but done, long overdue, not past deadline, R": {R, F, onTheHour, longDead, T, F, weekAfterTheHour(), F, F, 0}, "prev ran but done, long overdue, not past deadline, F": {f, F, onTheHour, longDead, T, F, weekAfterTheHour(), F, F, 0}, "prev ran but done, long overdue, no deadline, A":       {A, F, onTheHour, noDead, T, F, weekAfterTheHour(), F, F, 0}, "prev ran but done, long overdue, no deadline, R":       {R, F, onTheHour, noDead, T, F, weekAfterTheHour(), F, F, 0}, "prev ran but done, long overdue, no deadline, F":       {f, F, onTheHour, noDead, T, F, weekAfterTheHour(), F, F, 0}, "prev ran but done, long overdue, past medium deadline, A": {A, F, onTheHour, mediumDead, T, F, weekAfterTheHour(), T, F, 1}, "prev ran but done, long overdue, past short deadline, A":  {A, F, onTheHour, shortDead, T, F, weekAfterTheHour(), T, F, 1}, "prev ran but done, long overdue, past medium deadline, R": {R, F, onTheHour, mediumDead, T, F, weekAfterTheHour(), T, F, 1}, "prev ran but done, long overdue, past short deadline, R":  {R, F, onTheHour, shortDead, T, F, weekAfterTheHour(), T, F, 1}, "prev ran but done, long overdue, past medium deadline, F": {f, F, onTheHour, mediumDead, T, F, weekAfterTheHour(), T, F, 1}, "prev ran but done, long overdue, past short deadline, F":  {f, F, onTheHour, shortDead, T, F, weekAfterTheHour(), T, F, 1}, </ins> } for name, tc := range testCases { sj := cronJob()

if err != nil { return starts, fmt.Errorf("Unparseable schedule: %s : %s", sj.Spec.Schedule, err) } var earliestTime time.Time if sj.Status.LastScheduleTime != nil { earliestTime = sj.Status.LastScheduleTime.Time

// CronJob as last known start time. earliestTime = sj.ObjectMeta.CreationTimestamp.Time } <ins> if sj.Spec.StartingDeadlineSeconds != nil { // Controller is not going to schedule anything below this point schedulingDeadline := now.Add(-time.Second * time.Duration(*sj.Spec.StartingDeadlineSeconds)) if schedulingDeadline.After(earliestTime) { earliestTime = schedulingDeadline } } </ins> if earliestTime.After(now) { return []time.Time{}, nil }

// but less than "lots". if len(starts) > 100 { // We can't get the most recent times so just return an empty slice <del> return []time.Time{}, fmt.Errorf("Too many missed start times to list") </del> <ins> return []time.Time{}, fmt.Errorf("Too many missed start time (> 100). Set or decrease .spec.startingDeadlineSeconds or check clock skew.") </ins> } } return starts, nil

} } { <del> // Case 6: now is way way ahead of last start time. </del> <ins> // Case 6: now is way way ahead of last start time, and there is no deadline. </ins> sj.ObjectMeta.CreationTimestamp = metav1.Time{Time: T1.Add(-2 * time.Hour)} sj.Status.LastScheduleTime = &metav1.Time{Time: T1.Add(-1 * time.Hour)} now := T2.Add(10 * 24 * time.Hour)

t.Errorf("unexpected lack of error") } } <ins> { // Case 7: now is way way ahead of last start time, but there is a short deadline. sj.ObjectMeta.CreationTimestamp = metav1.Time{Time: T1.Add(-2 * time.Hour)} sj.Status.LastScheduleTime = &metav1.Time{Time: T1.Add(-1 * time.Hour)} now := T2.Add(10 * 24 * time.Hour) // Deadline is short deadline := int64(2 * 60 * 60) sj.Spec.StartingDeadlineSeconds = &deadline _, err := getRecentUnmetScheduleTimes(sj, now) if err != nil { t.Errorf("unexpected error") } } </ins> }