Repository: kubernetes-sigs/kueue

Issue:
Kueue controller is trying to access third party GVKs which doesn't have access to
**What happened**:

I'm trying to use Kueue with Kserve for model inference (through the Deployment and Pod integration) and I'm seeing many error logs in the Kueue controller manager like:

```
W0512 16:37:49.006396       1 reflector.go:569] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:108: failed to list *v1.PartialObjectMetadata: inferenceservices.serving.kserve.io is forbidden: User "system:serviceaccount:kueue-system:kueue-controller-manager" cannot list resource "inferenceservices" in API group "serving.kserve.io" at the cluster scope
E0512 16:37:49.006442       1 reflector.go:166] "Unhandled Error" err="sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:108: Failed to watch *v1.PartialObjectMetadata: failed to list *v1.PartialObjectMetadata: inferenceservices.serving.kserve.io is forbidden: User \"system:serviceaccount:kueue-system:kueue-controller-manager\" cannot list resource \"inferenceservices\" in API group \"serving.kserve.io\" at the cluster scope" logger="UnhandledError"
```

**What you expected to happen**:

No error logs.

**How to reproduce it (as minimally and precisely as possible)**:

1. Clone this repo https://github.com/pierDipi/kueue-access-unknown-gvks
    `git clone git@github.com:pierDipi/kueue-access-unknown-gvks.git`
1. Run `kind create cluster`
2. Run `./repro.sh`
3. See error logs: `kubectl logs -n kueue-system -l=control-plane=controller-manager -f`

**Anything else we need to know?**: I don't want to give Kueue access to a resource that doesn't need to have access to.

**Environment**: KinD
- Kubernetes version (use `kubectl version`): 
  ```
  $ kubectl version
  # ...
  Server Version: v1.32.2
  ```
- Kueue version (use `git describe --tags --dirty --always`): 0.11.4 (latest as of today)
- Cloud provider or hardware configuration: N/A
- OS (e.g: `cat /etc/os-release`): Linux (Fedora)
- Kernel (e.g. `uname -a`): N/A

Hints from issue comments:
@mimowo: This is probably related to the recent updates in [FindAncestorJobManagedByKueue](https://github.com/kubernetes-sigs/kueue/blob/0db959e83edda267222125cbca98ab1f41c796ac/pkg/controller/jobframework/reconciler.go#L691). 

The intention was two be able to support AppWrapper and be able to traverse the ownership chain without requiring management of the CRDs on the way. 

I'm wondering if we could patch this just by checking in advance if Kueue has the required access to read the CRD. If it does not have, then it will not work anyway.

cc @dgrove-oss  @tenzen-y @mbobrovskyi @dgrove-oss: We probably should patch it by catching the error, realizing we don't have read access, logging it at the debug level, and not returning an error to the controller runtime.

It's not directly relevant, but I did notice that in the [provided reproducer yaml,](https://github.com/pierDipi/kueue-access-unknown-gvks/blob/9626645572a7990b12f771d00f61955ea1422027/500-isvc.yaml#L6-#L20) the `InferenceService` has a queue-name label.  If Kueue isn't allowed to read InferenceService, that label is not going to be useful.

@mimowo: > We probably should patch it by catching the error, realizing we don't have read access, logging it at the debug level, and not returning an error to the controller runtime.

right this would work but realizing we don't have read access may require parsing the error which isn't clean, it might be a necessity. Im wondering if there is some API/function which would allow to check us if we have the read access prior to the attempt of reading.@pierDipi: > It's not directly relevant, but I did notice that in the [provided reproducer yaml,](https://github.com/pierDipi/kueue-access-unknown-gvks/blob/9626645572a7990b12f771d00f61955ea1422027/500-isvc.yaml#L6-#L20) the `InferenceService` has a queue-name label. If Kueue isn't allowed to read InferenceService, that label is not going to be useful.

As a note, the label is at the InferenceService level because we leverage Kserve labels propagation to propagete the Queue label to the Deployment/Pods.@pierDipi: > Im wondering if there is some API/function which would allow to check us if we have the read access prior to the attempt of reading

We can use `SelfSubjectAccessReview` API with get, list, watch verbs?
- https://kubernetes.io/docs/reference/access-authn-authz/authorization/#checking-api-access
- https://kubernetes.io/docs/reference/kubernetes-api/authorization-resources/self-subject-access-review-v1/@mimowo: Looks promising, we would need to check how this can be used by a go-client in Kueue code.@kannon92: I think this is another example of this error manifesting itself:

https://github.com/kubernetes-sigs/kueue/issues/4141#issuecomment-2877537125@mimowo: /assign 
Tentatively, as this is an important regression. 

However, I'm loaded with other release-related tasks, so if there is someone who is willing to look into this, then be my guest :)@qti-haeyoon: Regardless of the planned patch (which is to check access before attempting to read), should users update the Kueue controller's [ClusterRole](https://github.com/kubernetes-sigs/kueue/blob/main/charts/kueue/templates/rbac/role.yaml) manifest if we use third party GVK?@mimowo: /kind regression
Since this was not an issue prior to 0.11.4.
@mimowo: > Regardless of the planned patch (which is to check access before attempting to read), should users update the Kueue controller's [ClusterRole](https://github.com/kubernetes-sigs/kueue/blob/main/charts/kueue/templates/rbac/role.yaml) manifest if we use third party GVK?

This is a good question. The answer will depend on the way we decide to fix it, but I would strive to preserve the previous behavior (before 0.11.4), so this is not needed. 

I believe the problematic PR is this: https://github.com/kubernetes-sigs/kueue/pull/4824, which tries to walk up the owner reference even outside of frameworks recognized by Kueue. This may be a nice feature, but I don't think this was required as a bugfix. This PR aimed to support AppWrapper properly it seems in cases where the interim framework is either custom or known but disabled.

However, the problem with that is we may walk into frameworks we don't have permission for. And checking permission is hard. Yes, we have `SelfSubjectAccessReview`, but this is actually a wrapper around a POST call to the API-server. I don't think we can afford sending these posts from our webhooks (and the traversal function is used in webhooks). Similarly, I don't think even capturing the error as suggested in https://github.com/kubernetes-sigs/kueue/issues/5235#issuecomment-2873659015 would not be feasible, because it means that controller-runtime repeatedly will send requests to API server to LIST the resource and fail, this seems like too much churn.

So, the way I would propose to fix the regression, and yet maintain the fix  https://github.com/kubernetes-sigs/kueue/pull/4824 would be to only walk up the ownership over known frameworks (enabled + disabled + listed in externalFrameworks). This way we make sure, that if an admin wants some custom framework to work with AppWrapper they can list it inside externalFrameworks. And to be fair, it is probably ok to only walk over enabled + externalFrameworks. Then users of AppWrapper who want to use it for disabled built-in integrations could add the integration into the list of externalFrameworks.

If this approach gets support it will not require changes to the permissions.

cc @tenzen-y @dgrove-oss @PBundyra @gabesaba let me know if you have some opinions.@tenzen-y: > And to be fair, it is probably ok to only walk over enabled + externalFrameworks. Then users of AppWrapper who want to use it for disabled built-in integrations could add the integration into the list of externalFrameworks.

Does this mean the following framework enabling and disabling?

```yaml
integration:
  frameworks:
  - "batch/job"
  - "workload.codeflare.dev/appwrapper"
  externalFrameworks:
  - "jobset.x-k8s.io/jobset"
```@mimowo: Yes, but I'm hesitant between two options, you posted like in (1.):

1. 
- only consider looking up for workload for enabled integrations + externalFrameworks
- walk up ownership only for enabled integrations + externalFrameworks (stop on disabled integrations)

2. 
- only consider looking up for workload for enabled integrations + externalFrameworks
- walk up ownership only for enabled and disabled built-in integrations + externalFrameworks

I think actually (2.) might be less work, and less breaking to admins. It should be ok, because for the disabled integrations we provide RBAC permissions with Kueue, so we can walk up.  This would be minimal fix to only exclude walking up the tree for unknown frameworks completely. 

So, I'm now leaning to (2). wdyt? 

EDIT: in other words, externalFrameworks would only be needed for CustomJobs wrapped by AppWrapper. Not for JobSet as Kueue knows it provides RBAC for JobSet.@dgrove-oss: My recollection of the situation we were trying to fix by doing this is that we have to walk through disabled integrations (eg, suppose Job integration is disabled, but JobSet is enabled).  But option 2 should work.... we look through any kind that Kueue is capable of managing (enabled or disabled), but stop when we find something we don't know about.  And it is simpler since we don't have to do a potentially complex subject access review.@tenzen-y: > I think actually (2.) might be less work, and less breaking to admins. It should be ok, because for the disabled integrations we provide RBAC permissions with Kueue, so we can walk up. This would be minimal fix to only exclude walking up the tree for unknown frameworks completely.
> 
> So, I'm now leaning to (2). wdyt?

I'm fine with (2). Today, we have StatefulSet and Deployment supports. The popular OSS projects often depend on those core resources, like Knative. I think (2) would work well.


Solution patch:
diff --git a/pkg/controller/jobframework/integrationmanager.go b/pkg/controller/jobframework/integrationmanager.go
index 812404a07..a2e0e3785 100644
--- a/pkg/controller/jobframework/integrationmanager.go
+++ b/pkg/controller/jobframework/integrationmanager.go
@@ -20,6 +20,7 @@ import (
 	"context"
 	"errors"
 	"fmt"
+	"maps"
 	"slices"
 	"sort"
 	"sync"
@@ -198,6 +199,23 @@ func (m *integrationManager) getList() []string {
 	return ret
 }
 
+func (m *integrationManager) isKnownOwner(ownerRef *metav1.OwnerReference) bool {
+	for _, cbs := range m.integrations {
+		if cbs.matchingOwnerReference(ownerRef) {
+			return true
+		}
+	}
+	for _, jt := range m.externalIntegrations {
+		if ownerReferenceMatchingGVK(ownerRef, jt.GetObjectKind().GroupVersionKind()) {
+			return true
+		}
+	}
+	// ReplicaSet is an interim owner from Pod to Deployment. We call it known
+	// so that the users don't need to list	it explicitly in their configs.
+	// Note that Kueue provides RBAC permissions allowing for traversal over it.
+	return ownerRef.Kind == "ReplicaSet" && ownerRef.APIVersion == "apps/v1"
+}
+
 func (m *integrationManager) getJobTypeForOwner(ownerRef *metav1.OwnerReference) runtime.Object {
 	for jobKey := range m.getEnabledIntegrations() {
 		cbs, found := m.integrations[jobKey]
@@ -286,6 +304,23 @@ func EnableIntegrationsForTest(tb testing.TB, names ...string) func() {
 	}
 }
 
+// EnableExternalIntegrationsForTest - should be used only in tests
+// Mark the frameworks identified by names and return a revert function.
+func EnableExternalIntegrationsForTest(tb testing.TB, names ...string) func() {
+	tb.Helper()
+	old := maps.Clone(manager.externalIntegrations)
+	for _, name := range names {
+		if err := manager.registerExternal(name); err != nil {
+			tb.Fatalf("failed to register external framework: %q", name)
+		}
+	}
+	return func() {
+		manager.mu.Lock()
+		manager.externalIntegrations = old
+		manager.mu.Unlock()
+	}
+}
+
 // GetIntegration looks-up the framework identified by name in the currently registered
 // list of frameworks returning its callbacks and true if found.
 func GetIntegration(name string) (IntegrationCallbacks, bool) {
diff --git a/pkg/controller/jobframework/reconciler.go b/pkg/controller/jobframework/reconciler.go
index 63b0e9055..a0e0b385f 100644
--- a/pkg/controller/jobframework/reconciler.go
+++ b/pkg/controller/jobframework/reconciler.go
@@ -715,9 +715,14 @@ func FindAncestorJobManagedByKueue(ctx context.Context, c client.Client, jobObj
 
 		owner := metav1.GetControllerOf(currentObj)
 		if owner == nil {
+			log.V(3).Info("stop walking up as the owner is not found", "owner", klog.KObj(currentObj))
 			return topLevelJob, nil
 		}
 
+		if !manager.isKnownOwner(owner) {
+			log.V(3).Info("stop walking up as the owner is not known", "owner", klog.KObj(currentObj))
+			return topLevelJob, nil
+		}
 		parentObj := GetEmptyOwnerObject(owner)
 		managed := parentObj != nil
 		if parentObj == nil {



Test patch:
diff --git a/pkg/controller/jobframework/reconciler_test.go b/pkg/controller/jobframework/reconciler_test.go
index 9b09c951d..0477c0cf8 100644
--- a/pkg/controller/jobframework/reconciler_test.go
+++ b/pkg/controller/jobframework/reconciler_test.go
@@ -24,12 +24,14 @@ import (
 	"github.com/google/go-cmp/cmp/cmpopts"
 	kfmpi "github.com/kubeflow/mpi-operator/pkg/apis/kubeflow/v2beta1"
 	awv1beta2 "github.com/project-codeflare/appwrapper/api/v1beta2"
+	appsv1 "k8s.io/api/apps/v1"
 	batchv1 "k8s.io/api/batch/v1"
 	corev1 "k8s.io/api/core/v1"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 	"k8s.io/apimachinery/pkg/types"
 	"k8s.io/utils/clock"
 	testingclock "k8s.io/utils/clock/testing"
+	"k8s.io/utils/ptr"
 	"sigs.k8s.io/controller-runtime/pkg/client"
 	"sigs.k8s.io/jobset/api/jobset/v1alpha2"
 
@@ -38,6 +40,7 @@ import (
 	"sigs.k8s.io/kueue/pkg/util/kubeversion"
 	utiltesting "sigs.k8s.io/kueue/pkg/util/testing"
 	testingaw "sigs.k8s.io/kueue/pkg/util/testingjobs/appwrapper"
+	testingdeployment "sigs.k8s.io/kueue/pkg/util/testingjobs/deployment"
 	testingjob "sigs.k8s.io/kueue/pkg/util/testingjobs/job"
 	"sigs.k8s.io/kueue/pkg/util/testingjobs/jobset"
 	testingmpijob "sigs.k8s.io/kueue/pkg/util/testingjobs/mpijob"
@@ -69,6 +72,7 @@ func TestFindAncestorJobManagedByKueue(t *testing.T) {
 	cases := map[string]struct {
 		manageJobsWithoutQueueName bool
 		integrations               []string
+		externalFrameworks         []string
 		ancestors                  []client.Object
 		job                        client.Object
 		wantManaged                client.Object
@@ -315,10 +319,80 @@ func TestFindAncestorJobManagedByKueue(t *testing.T) {
 				Obj(),
 			wantManaged: testingaw.MakeAppWrapper("aw", jobNamespace).UID("aw").Obj(),
 		},
+		"Job -> CronJob (external framework, not enabled) -> AppWrapper (queue-name) => AppWrapper": {
+			integrations: []string{"workload.codeflare.dev/appwrapper"},
+			ancestors: []client.Object{
+				testingaw.MakeAppWrapper("aw", jobNamespace).UID("aw").Queue("test-q").Obj(),
+				&batchv1.CronJob{
+					ObjectMeta: metav1.ObjectMeta{
+						Name:      "cronjob",
+						Namespace: jobNamespace,
+						OwnerReferences: []metav1.OwnerReference{{
+							Name:       "aw",
+							APIVersion: "workload.codeflare.dev/appwrapper",
+							Kind:       "AppWrapper",
+							UID:        "aw",
+							Controller: ptr.To(true),
+						}},
+					},
+				},
+			},
+			job: testingjob.MakeJob("job", jobNamespace).UID("job").
+				OwnerReference("cronjob", batchv1.SchemeGroupVersion.WithKind("CronJob")).
+				Obj(),
+		},
+		"Job -> CronJob (external framework, enabled) -> AppWrapper (queue-name) => AppWrapper": {
+			integrations:       []string{"workload.codeflare.dev/appwrapper"},
+			externalFrameworks: []string{"CronJob.v1.batch"},
+			ancestors: []client.Object{
+				testingaw.MakeAppWrapper("aw", jobNamespace).UID("aw").Queue("test-q").Obj(),
+				&batchv1.CronJob{
+					ObjectMeta: metav1.ObjectMeta{
+						Name:      "cronjob",
+						Namespace: jobNamespace,
+						OwnerReferences: []metav1.OwnerReference{{
+							Name:       "aw",
+							APIVersion: "workload.codeflare.dev/v1beta2",
+							Kind:       "AppWrapper",
+							UID:        "aw",
+							Controller: ptr.To(true),
+						}},
+					},
+				},
+			},
+			job: testingjob.MakeJob("job", jobNamespace).UID("job").
+				OwnerReference("cronjob", batchv1.SchemeGroupVersion.WithKind("CronJob")).
+				Obj(),
+			wantManaged: testingaw.MakeAppWrapper("aw", jobNamespace).UID("aw").Queue("test-q").Obj(),
+		},
+		"Pod -> ReplicaSet -> Deployment (queue-name) => Deployment": {
+			integrations: []string{"pod", "deployment"},
+			ancestors: []client.Object{
+				testingdeployment.MakeDeployment("deploy", jobNamespace).UID("deploy").Queue("test-q").Obj(),
+				&appsv1.ReplicaSet{
+					ObjectMeta: metav1.ObjectMeta{
+						Name:      "rs",
+						Namespace: jobNamespace,
+						OwnerReferences: []metav1.OwnerReference{{
+							Name:       "deploy",
+							APIVersion: "apps/v1",
+							Kind:       "Deployment",
+							UID:        "deploy",
+							Controller: ptr.To(true),
+						}},
+					},
+				},
+			},
+			job: testingjob.MakeJob("pod", jobNamespace).UID("pod").
+				OwnerReference("rs", appsv1.SchemeGroupVersion.WithKind("ReplicaSet")).
+				Obj(),
+			wantManaged: testingdeployment.MakeDeployment("deploy", jobNamespace).UID("deploy").Queue("test-q").Obj(),
+		},
 	}
 	for name, tc := range cases {
 		t.Run(name, func(t *testing.T) {
 			t.Cleanup(EnableIntegrationsForTest(t, tc.integrations...))
+			t.Cleanup(EnableExternalIntegrationsForTest(t, tc.externalFrameworks...))
 			ctx, _ := utiltesting.ContextWithLog(t)
 			recorder := &utiltesting.EventRecorder{}
 			builder := utiltesting.NewClientBuilder(kfmpi.AddToScheme, awv1beta2.AddToScheme, v1alpha2.AddToScheme)
diff --git a/pkg/controller/jobs/job/job_webhook_test.go b/pkg/controller/jobs/job/job_webhook_test.go
index c00cb1b0a..15f24687b 100644
--- a/pkg/controller/jobs/job/job_webhook_test.go
+++ b/pkg/controller/jobs/job/job_webhook_test.go
@@ -700,7 +700,6 @@ func TestDefault(t *testing.T) {
 				OwnerReference("owner", jobsetapi.SchemeGroupVersion.WithKind("JobSet")).
 				Queue("default").
 				Obj(),
-			wantErr: jobframework.ErrWorkloadOwnerNotFound,
 		},
 	}
 	for name, tc := range testcases {
diff --git a/test/e2e/singlecluster/e2e_test.go b/test/e2e/singlecluster/e2e_test.go
index 530725152..68a7e6419 100644
--- a/test/e2e/singlecluster/e2e_test.go
+++ b/test/e2e/singlecluster/e2e_test.go
@@ -17,10 +17,13 @@ limitations under the License.
 package e2e
 
 import (
+	"time"
+
 	"github.com/onsi/ginkgo/v2"
 	"github.com/onsi/gomega"
 	batchv1 "k8s.io/api/batch/v1"
 	corev1 "k8s.io/api/core/v1"
+	"k8s.io/apimachinery/pkg/api/resource"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 	"k8s.io/apimachinery/pkg/types"
 	"k8s.io/utils/ptr"
@@ -118,6 +121,85 @@ var _ = ginkgo.Describe("Kueue", func() {
 			util.ExpectAllPodsInNamespaceDeleted(ctx, k8sClient, ns)
 		})
 
+		ginkgo.It("Should allow to schedule Jobs via CronJob", func() {
+			cronJob := &batchv1.CronJob{
+				ObjectMeta: metav1.ObjectMeta{
+					Name:      "test-cronjob",
+					Namespace: ns.Name,
+				},
+				Spec: batchv1.CronJobSpec{
+					Schedule:          "* * * * *",
+					ConcurrencyPolicy: batchv1.ForbidConcurrent,
+					JobTemplate: batchv1.JobTemplateSpec{
+						ObjectMeta: metav1.ObjectMeta{
+							Labels: map[string]string{
+								constants.QueueLabel: localQueue.Name,
+							},
+						},
+						Spec: batchv1.JobSpec{
+							Template: corev1.PodTemplateSpec{
+								Spec: corev1.PodSpec{
+									RestartPolicy: corev1.RestartPolicyNever,
+									Containers: []corev1.Container{
+										{
+											Name:    "c",
+											Image:   util.E2eTestAgnHostImage,
+											Command: util.BehaviorExitFast,
+											Resources: corev1.ResourceRequirements{
+												Requests: corev1.ResourceList{
+													corev1.ResourceCPU: resource.MustParse("1"),
+												},
+												Limits: corev1.ResourceList{
+													corev1.ResourceCPU: resource.MustParse("1"),
+												},
+											},
+										},
+									},
+								},
+							},
+						},
+					},
+				},
+			}
+			util.MustCreate(ctx, k8sClient, cronJob)
+			ginkgo.DeferCleanup(func() {
+				gomega.Expect(util.DeleteAllCronJobsInNamespace(ctx, k8sClient, ns)).Should(gomega.Succeed())
+			})
+
+			ginkgo.By("Patch the last start time to be in the past so that it starts immediately", func() {
+				gomega.Eventually(func(g gomega.Gomega) {
+					g.Expect(k8sClient.Get(ctx, client.ObjectKeyFromObject(cronJob), cronJob)).To(gomega.Succeed())
+					nextSchedule := cronJob.CreationTimestamp.Add(-2 * time.Minute)
+					cronJob.Status.LastScheduleTime = ptr.To(metav1.Time{Time: nextSchedule})
+					g.Expect(k8sClient.Status().Update(ctx, cronJob)).Should(gomega.Succeed())
+				}, util.LongTimeout, util.Interval).Should(gomega.Succeed())
+			})
+
+			createJobs := &batchv1.JobList{}
+			ginkgo.By("Check that the Job is create and retrieve it", func() {
+				gomega.Eventually(func(g gomega.Gomega) {
+					g.Expect(k8sClient.List(ctx, createJobs, client.InNamespace(ns.Name))).To(gomega.Succeed())
+					g.Expect(createJobs.Items).To(gomega.HaveLen(1))
+				}, util.LongTimeout, util.Interval).Should(gomega.Succeed())
+			})
+
+			createdJob := createJobs.Items[0]
+			ginkgo.By("verify the job has the nodeSelector assigned", func() {
+				jobKey := client.ObjectKeyFromObject(&createdJob)
+				util.ExpectJobUnsuspendedWithNodeSelectors(ctx, k8sClient, jobKey, map[string]string{
+					"instance-type": "on-demand",
+				})
+			})
+			ginkgo.By("verify the workload was created and admitted for the Job", func() {
+				createdWorkload := &kueue.Workload{}
+				wlLookupKey := types.NamespacedName{Name: workloadjob.GetWorkloadNameForJob(createdJob.Name, createdJob.UID), Namespace: ns.Name}
+				gomega.Eventually(func(g gomega.Gomega) {
+					g.Expect(k8sClient.Get(ctx, wlLookupKey, createdWorkload)).Should(gomega.Succeed())
+					g.Expect(workload.HasQuotaReservation(createdWorkload)).Should(gomega.BeTrue())
+				}, util.LongTimeout, util.Interval).Should(gomega.Succeed())
+			})
+		})
+
 		ginkgo.It("Should unsuspend a job and set nodeSelectors", func() {
 			// Use a binary that ends.
 			sampleJob = (&testingjob.JobWrapper{Job: *sampleJob}).Image(util.E2eTestAgnHostImage, util.BehaviorExitFast).Obj()
diff --git a/test/util/util.go b/test/util/util.go
index 476491822..2e1022fd5 100644
--- a/test/util/util.go
+++ b/test/util/util.go
@@ -158,6 +158,10 @@ func DeleteNamespace(ctx context.Context, c client.Client, ns *corev1.Namespace)
 	return nil
 }
 
+func DeleteAllCronJobsInNamespace(ctx context.Context, c client.Client, ns *corev1.Namespace) error {
+	return deleteAllObjectsInNamespace(ctx, c, ns, &batchv1.CronJob{})
+}
+
 func DeleteAllJobsInNamespace(ctx context.Context, c client.Client, ns *corev1.Namespace) error {
 	return deleteAllObjectsInNamespace(ctx, c, ns, &batchv1.Job{})
 }