Change:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --This started happening when we upgraded to Kubernetes 1.21. The following message occurs: The Cronjob is pretty simple: No error. But in fact, it does appear jobs ARE being removed properly regardless of the error message. Use the above. We tried deleting and recreating the cronjob both with and and neither seems to work. Note the API version was at batch/v1beta1 before we upgraded, and we did change it to batch/v1 to squelch the deprecation notice. Kubernetes version (use ):Cloud provider or hardware configuration: AWS OS (e.g: ): From nodes (master is identical):Kernel (e.g. ): Install tools: Kops Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig apps
i think it's due to the upgrade of your cluster. The error means resource version can not match? what's your cluster version before? your new created cronjob is working well, right? like below:
The previous cluster version was 1.18, and yes I know we did a bad thing by skipping two minor releases. The newly created cronjob is working fine. And strangely, there are no old jobs that spawned - except for the three most recently run, all of them obviously created in the new cluster (we use the defaults here). So jobs ARE getting cleaned up. That's what makes the error messages baffling.
is set to default 3, is set to default 1. see for details.
It's a bug of controllerv2 for cronjob. It passes a cronjob with old resourceVersion to update its status when cleaning up jobs. The CronJobControllerV2 feature flag graduates to beta and set to be enabled by default in v1.21. So , this started happening when you upgraded to Kubernetes 1.21. I will try to submit a PR to fix it. /assign
We are also seeing this error message in our cloudwatch logs. Some of our cronjobs say that there's an active job running, yet the job referenced has already been completed. Could that be a result of this bug? For example: As you can see it successfully completed. Yet the cronjob lists it as active: Relevant Cloudwatch logs:
seeing the same thing on GKE 1.21.5-gke.1302
hi! any news on that PR? it seems there is a need for rebase
The whole thing with JobAlreadyActive is that it prevents next schedules of a Job, since we have as our main mechanism to prevent overlapping.
Thanks for reminding me. I've updated the PR.
can you please confirm, that this bug is what causing CronJobs to stuck on JobAlreadyActive reason?
I‘m not sure whether it is also a result of this bug since I didn't reproduce that as you described. But according to the code and logs, they seem to be related. Maybe you can retest it when this fixed.
OK, I'll try to take a closer look at it recently.
we have the same thing with Forbid we use helm templates to deploy using Spinnaker so everytime we recreate cronjobs they have the old . I think they automatically get modified when cronjob objects are created to upgrade to I didn't know that objects were automatically modified when they have the old apiVersion. I noticed Ingress objects were also modified. so I'm updating our helm charts to fix this
/cc this is related to cronjob v2 controller
I wrote a bash script to clean up the jobs affected by this bug it looks for active cronjobs then their active jobs checks the actual status of the jobs deletes the jobs that are complete
the fix is merged and backported to 1.21; are you able to use the latest version to test the fix? It is available in 1.21.9, 1.22.6 and 1.23.3
thanks for confirming the patch versions, I wasn't sure what patch version it would come in. we're on the latest version of EKS (1.21.5). we'll need to wait for it to come to EKS

Discussion:
} cj.Status.Active = append(cj.Status.Active, *jobRef) cj.Status.LastScheduleTime = &metav1.Time{Time: *scheduledTime} <del> if _, err := jm.cronJobControl.UpdateStatus(ctx, cj); err != nil { </del> <ins> updatedCJ, err = jm.cronJobControl.UpdateStatus(ctx, cj) if err != nil { </ins> klog.InfoS("Unable to update status", "cronjob", klog.KRef(cj.GetNamespace(), cj.GetName()), "resourceVersion", cj.ResourceVersion, "err", err) return cj, nil, fmt.Errorf("unable to update status for %s (rv = %s): %v", klog.KRef(cj.GetNamespace(), cj.GetName()), cj.ResourceVersion, err) } t := nextScheduledTimeDuration(sched, now) <del> return cj, t, nil </del> <ins> return updatedCJ, t, nil </ins> } func getJobName(cj *batchv1.CronJob, scheduledTime time.Time) string {

batchv1 "k8s.io/api/batch/v1" corev1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" <ins> "k8s.io/apimachinery/pkg/util/runtime" </ins> "k8s.io/apimachinery/pkg/util/wait" "k8s.io/client-go/informers" clientset "k8s.io/client-go/kubernetes"

ns := framework.CreateTestingNamespace(namespaceName, server, t) defer framework.DeleteTestingNamespace(ns, server, t) <ins> backupHandlers := runtime.ErrorHandlers runtime.ErrorHandlers = append(runtime.ErrorHandlers, func(e error) { t.Fatalf("Failed with error: %v", e) }) defer func() { runtime.ErrorHandlers = backupHandlers }() </ins> cjClient := clientSet.BatchV1().CronJobs(ns.Name) stopCh := make(chan struct{})