Repository: kubernetes-sigs/kueue
Pull request #114 (2022-03-11 12:32:07)

Enhance Kueue.Queue.Status test suite
Signed-off-by: Carlos Eduardo Arango Gutierrez <carangog@redhat.com>

<!--  Thanks for sending a pull request!  Here are some tips for you:

1. If this is your first time, please read our contributor guidelines: https://git.k8s.io/community/contributors/guide/first-contribution.md#your-first-contribution and developer guide https://git.k8s.io/community/contributors/devel/development.md#development-guide
2. Please label this pull request according to what type of issue you are addressing, especially if this is a release targeted pull request. For reference on required PR/issue labels, read here:
https://git.k8s.io/community/contributors/devel/sig-release/release.md#issuepr-kind-label
3. Ensure you have added or ran the appropriate tests for your PR: https://git.k8s.io/community/contributors/devel/sig-testing/testing.md
4. If you want *faster* PR reviews, read how: https://git.k8s.io/community/contributors/guide/pull-requests.md#best-practices-for-faster-reviews
5. If the PR is unfinished, see how to mark it: https://git.k8s.io/community/contributors/guide/pull-requests.md#marking-unfinished-pull-requests
-->

#### What type of PR is this?
/kind cleanup

<!--
Add one of the following kinds:
/kind bug
/kind cleanup
/kind documentation
/kind feature

Optionally add one or more of the following kinds if applicable:
/kind api-change
/kind deprecation
/kind failing-test
/kind flake
/kind regression
-->

#### What this PR does / why we need it:

#### Which issue(s) this PR fixes:
<!--
*Automatically closes linked issue when PR is merged.
Usage: `Fixes #<issue number>`, or `Fixes (paste link of issue)`.
_If PR is about `failing-tests or flakes`, please post the related issues/tests in a comment and do not use `Fixes`_*
-->
Fixes #

#### Special notes for your reviewer:



k8s-ci-robot: [APPROVALNOTIFIER] This PR is **NOT APPROVED**

This pull-request has been approved by: *<a href="https://github.com/kubernetes-sigs/kueue/pull/114#" title="Author self-approved">ArangoGutierrez</a>*
To complete the [pull request process](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process), please assign **alculquicondor** after the PR has been reviewed.
You can assign the PR to them by writing `/assign @alculquicondor` in a comment when ready.

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=kubernetes-sigs%2Fkueue).

<details open>
Needs approval from an approver in each of these files:

- **[OWNERS](https://github.com/kubernetes-sigs/kueue/blob/main/OWNERS)**

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":["alculquicondor"]} -->
ArangoGutierrez: /assign @alculquicondor
ArangoGutierrez: /retest-required
ArangoGutierrez: Previous test failure:
```bash
Test started today at 7:32 AM failed after 15m0s. ([less info](https://prow.k8s.io/spyglass/lens/metadata/iframe?req=%7B%22artifacts%22%3A%5B%22finished.json%22%2C%22podinfo.json%22%2C%22prowjob.json%22%2C%22started.json%22%5D%2C%22index%22%3A0%2C%22src%22%3A%22gs%2Fkubernetes-jenkins%2Fpr-logs%2Fpull%2Fkubernetes-sigs_kueue%2F114%2Fpull-kueue-build-image-main%2F1502261086198435840%22%7D&topURL=https%3A//prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_kueue/114/pull-kueue-build-image-main/1502261086198435840&lensIndex=0#))

Init container clonerefs not ready: (state: waiting, reason: "PodInitializing", message: "") Init container initupload not ready: (state: waiting, reason: "PodInitializing", message: "") Init container place-entrypoint not ready: (state: waiting, reason: "PodInitializing", message: "")
```
k8s-ci-robot: [APPROVALNOTIFIER] This PR is **NOT APPROVED**

This pull-request has been approved by: *<a href="https://github.com/kubernetes-sigs/kueue/pull/114#" title="Author self-approved">ArangoGutierrez</a>*
To complete the [pull request process](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process), please ask for approval from **alculquicondor** after the PR has been reviewed.

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=kubernetes-sigs%2Fkueue).

<details open>
Needs approval from an approver in each of these files:

- **[OWNERS](https://github.com/kubernetes-sigs/kueue/blob/main/OWNERS)**

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":["alculquicondor"]} -->
k8s-ci-robot: @ArangoGutierrez: The following test **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-kueue-verify-main | 4971e0c25be013194e6098ddb561e6d925fe1781 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_kueue/114/pull-kueue-verify-main/1502367507644682240) | true | `/test pull-kueue-verify-main`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=kueue&pr=114). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3AArangoGutierrez). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/kueue/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @ArangoGutierrez: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-kueue-verify-main | 4971e0c25be013194e6098ddb561e6d925fe1781 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_kueue/114/pull-kueue-verify-main/1502367507644682240) | true | `/test pull-kueue-verify-main`
pull-kueue-test-unit-main | 4971e0c25be013194e6098ddb561e6d925fe1781 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_kueue/114/pull-kueue-test-unit-main/1502367507573379072) | true | `/test pull-kueue-test-unit-main`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=kueue&pr=114). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3AArangoGutierrez). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/kueue/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @ArangoGutierrez: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-kueue-verify-main | 4971e0c25be013194e6098ddb561e6d925fe1781 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_kueue/114/pull-kueue-verify-main/1502367507644682240) | true | `/test pull-kueue-verify-main`
pull-kueue-test-unit-main | 4971e0c25be013194e6098ddb561e6d925fe1781 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_kueue/114/pull-kueue-test-unit-main/1502367507573379072) | true | `/test pull-kueue-test-unit-main`
pull-kueue-build-image-main | 4971e0c25be013194e6098ddb561e6d925fe1781 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_kueue/114/pull-kueue-build-image-main/1502367507682430976) | true | `/test pull-kueue-build-image-main`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=kueue&pr=114). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3AArangoGutierrez). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/kueue/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->

Patch:
diff --git a/api/v1alpha1/queue_types.go b/api/v1alpha1/queue_types.go
index 9e7f3e49b..94b175fa2 100644
--- a/api/v1alpha1/queue_types.go
+++ b/api/v1alpha1/queue_types.go
@@ -33,6 +33,7 @@ type ClusterQueueReference string
 type QueueStatus struct {
 	// PendingWorkloads is the number of workloads currently admitted to this
 	// queue not yet admitted to a ClusterQueue.
+	// +optional
 	PendingWorkloads int32 `json:"pendingWorkloads"`
 }
 
diff --git a/config/crd/bases/kueue.x-k8s.io_queues.yaml b/config/crd/bases/kueue.x-k8s.io_queues.yaml
index 743b1ccae..cd468bf46 100644
--- a/config/crd/bases/kueue.x-k8s.io_queues.yaml
+++ b/config/crd/bases/kueue.x-k8s.io_queues.yaml
@@ -48,8 +48,6 @@ spec:
                   admitted to this queue not yet admitted to a ClusterQueue.
                 format: int32
                 type: integer
-            required:
-            - pendingWorkloads
             type: object
         type: object
     served: true
diff --git a/pkg/controller/core/queue_controller.go b/pkg/controller/core/queue_controller.go
index 471ff9bf0..ecdf4184d 100644
--- a/pkg/controller/core/queue_controller.go
+++ b/pkg/controller/core/queue_controller.go
@@ -64,14 +64,16 @@ func (r *QueueReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl
 	ctx = ctrl.LoggerInto(ctx, log)
 	log.V(2).Info("Reconciling Queue")
 
-	pendingJobs, err := r.queues.Status(&queueObj)
+	// Shallow copy enough for now.
+	oldStatus := queueObj.Status
+
+	pending, err := r.queues.PendingWorkloads(&queueObj)
 	if err != nil {
 		r.log.Error(err, "Failed to retrieve queue status")
 		return ctrl.Result{}, err
 	}
 
-	oldStatus := queueObj.Status
-	queueObj.Status.PendingWorkloads = int32(pendingJobs)
+	queueObj.Status.PendingWorkloads = pending
 	if !equality.Semantic.DeepEqual(oldStatus, queueObj.Status) {
 		err := r.client.Status().Update(ctx, &queueObj)
 		return ctrl.Result{}, client.IgnoreNotFound(err)
diff --git a/pkg/queue/manager.go b/pkg/queue/manager.go
index 855b5885c..d5e24d1ad 100644
--- a/pkg/queue/manager.go
+++ b/pkg/queue/manager.go
@@ -36,9 +36,9 @@ const (
 )
 
 var (
-	queueDoesNotExistErr         = errors.New("queue doesn't exist")
-	clusterQueueDoesNotExistErr  = errors.New("clusterQueue doesn't exist")
-	clusterQueueAlreadyExistsErr = errors.New("clusterQueue already exists")
+	errQueueDoesNotExist         = errors.New("queue doesn't exist")
+	errClusterQueueDoesNotExist  = errors.New("clusterQueue doesn't exist")
+	errClusterQueueAlreadyExists = errors.New("clusterQueue already exists")
 )
 
 type Manager struct {
@@ -65,7 +65,7 @@ func (m *Manager) AddClusterQueue(ctx context.Context, cq *kueue.ClusterQueue) e
 	defer m.Unlock()
 
 	if _, ok := m.clusterQueues[cq.Name]; ok {
-		return clusterQueueAlreadyExistsErr
+		return errClusterQueueAlreadyExists
 	}
 	cqImpl := newClusterQueue(cq)
 	m.clusterQueues[cq.Name] = cqImpl
@@ -99,7 +99,7 @@ func (m *Manager) UpdateClusterQueue(cq *kueue.ClusterQueue) error {
 	defer m.Unlock()
 	cqImpl, ok := m.clusterQueues[cq.Name]
 	if !ok {
-		return clusterQueueDoesNotExistErr
+		return errClusterQueueDoesNotExist
 	}
 	// TODO(#8): recreate heap based on a change of queueing policy.
 	cqImpl.update(cq)
@@ -152,7 +152,7 @@ func (m *Manager) UpdateQueue(q *kueue.Queue) error {
 	defer m.Unlock()
 	qImpl, ok := m.queues[Key(q)]
 	if !ok {
-		return queueDoesNotExistErr
+		return errQueueDoesNotExist
 	}
 	if qImpl.ClusterQueue != string(q.Spec.ClusterQueue) {
 		oldCQ := m.clusterQueues[qImpl.ClusterQueue]
@@ -183,13 +183,16 @@ func (m *Manager) DeleteQueue(q *kueue.Queue) {
 	delete(m.queues, key)
 }
 
-func (m *Manager) Status(q *kueue.Queue) (int, error) {
+func (m *Manager) PendingWorkloads(q *kueue.Queue) (int32, error) {
 	m.RLock()
 	defer m.RUnlock()
-	if q := m.queues[Key(q)]; q != nil {
-		return len(q.items), nil
+
+	qImpl, ok := m.queues[Key(q)]
+	if !ok {
+		return 0, errQueueDoesNotExist
 	}
-	return 0, queueDoesNotExistErr
+
+	return int32(len(qImpl.items)), nil
 }
 
 // AddOrUpdateWorkload adds or updates workload to the corresponding queue.


Test patch:
diff --git a/pkg/queue/manager_test.go b/pkg/queue/manager_test.go
index acb6ce407..a4ccc0903 100644
--- a/pkg/queue/manager_test.go
+++ b/pkg/queue/manager_test.go
@@ -272,7 +272,15 @@ func TestStatus(t *testing.T) {
 			},
 			Spec: kueue.QueuedWorkloadSpec{QueueName: "foo"},
 		},
+		{
+			ObjectMeta: metav1.ObjectMeta{
+				Name:              "d",
+				CreationTimestamp: metav1.NewTime(now),
+			},
+			Spec: kueue.QueuedWorkloadSpec{QueueName: "foo"},
+		},
 	}
+
 	manager := NewManager(fake.NewClientBuilder().WithScheme(scheme).Build())
 	for _, q := range queues {
 		if err := manager.AddQueue(ctx, &q); err != nil {
@@ -286,12 +294,12 @@ func TestStatus(t *testing.T) {
 
 	cases := map[string]struct {
 		queue      *kueue.Queue
-		wantStatus int
+		wantStatus int32
 		wantErr    error
 	}{
 		"foo": {
 			queue:      &queues[0],
-			wantStatus: 2,
+			wantStatus: 3,
 			wantErr:    nil,
 		},
 		"bar": {
@@ -302,12 +310,12 @@ func TestStatus(t *testing.T) {
 		"fake": {
 			queue:      &kueue.Queue{ObjectMeta: metav1.ObjectMeta{Name: "fake"}},
 			wantStatus: 0,
-			wantErr:    queueDoesNotExistErr,
+			wantErr:    errQueueDoesNotExist,
 		},
 	}
 	for name, tc := range cases {
 		t.Run(name, func(t *testing.T) {
-			status, err := manager.Status(tc.queue)
+			status, err := manager.PendingWorkloads(tc.queue)
 			if err != tc.wantErr {
 				t.Errorf("Should have failed with: %s", err)
 			}
diff --git a/test/integration/controller/core/queue_controller_test.go b/test/integration/controller/core/queue_controller_test.go
index c8fe4500a..3a127d050 100644
--- a/test/integration/controller/core/queue_controller_test.go
+++ b/test/integration/controller/core/queue_controller_test.go
@@ -33,7 +33,7 @@ import (
 var _ = ginkgo.Describe("Queue controller", func() {
 	var (
 		ns           *corev1.Namespace
-		Queue        *kueue.Queue
+		queue        *kueue.Queue
 		clusterQueue *kueue.ClusterQueue
 	)
 
@@ -51,26 +51,26 @@ var _ = ginkgo.Describe("Queue controller", func() {
 			Resource(testing.MakeResource(resourceGPU).
 				Flavor(testing.MakeFlavor(flavorModelA, "5").Ceiling("10").Obj()).
 				Flavor(testing.MakeFlavor(flavorModelB, "5").Ceiling("10").Obj()).Obj()).Obj()
-		Queue = testing.MakeQueue("queue", ns.Name).ClusterQueue("cluster-queue").Obj()
-		gomega.Expect(k8sClient.Create(ctx, Queue)).To(gomega.Succeed())
+		queue = testing.MakeQueue("queue", ns.Name).ClusterQueue(clusterQueue.Name).Obj()
+		gomega.Expect(k8sClient.Create(ctx, queue)).To(gomega.Succeed())
 	})
 
 	ginkgo.AfterEach(func() {
-		gomega.Expect(framework.DeleteQueue(ctx, k8sClient, Queue)).To(gomega.Succeed())
+		gomega.Expect(framework.DeleteQueue(ctx, k8sClient, queue)).To(gomega.Succeed())
 		gomega.Expect(framework.DeleteClusterQueue(ctx, k8sClient, clusterQueue)).To(gomega.Succeed())
 	})
 
 	ginkgo.It("Should update status when workloads are created", func() {
 		workloads := []*kueue.QueuedWorkload{
 			testing.MakeQueuedWorkload("one", ns.Name).
-				Queue(Queue.Name).
-				Request(corev1.ResourceCPU, "2").Request(resourceGPU, "1").Obj(),
+				Queue(queue.Name).
+				Request(corev1.ResourceCPU, "2").Obj(),
 			testing.MakeQueuedWorkload("two", ns.Name).
-				Queue(Queue.Name).
-				Request(corev1.ResourceCPU, "3").Request(resourceGPU, "1").Obj(),
+				Queue(queue.Name).
+				Request(corev1.ResourceCPU, "3").Obj(),
 			testing.MakeQueuedWorkload("three", ns.Name).
-				Queue(Queue.Name).
-				Request(corev1.ResourceCPU, "1").Request(resourceGPU, "1").Obj(),
+				Queue(queue.Name).
+				Request(corev1.ResourceCPU, "1").Obj(),
 		}
 
 		ginkgo.By("Creating workloads")
@@ -79,10 +79,22 @@ var _ = ginkgo.Describe("Queue controller", func() {
 		}
 		gomega.Eventually(func() kueue.QueueStatus {
 			var updatedQueue kueue.Queue
-			gomega.Expect(k8sClient.Get(ctx, client.ObjectKeyFromObject(Queue), &updatedQueue)).To(gomega.Succeed())
+			gomega.Expect(k8sClient.Get(ctx, client.ObjectKeyFromObject(queue), &updatedQueue)).To(gomega.Succeed())
 			return updatedQueue.Status
 		}, framework.Timeout, framework.Interval).Should(testing.Equal(kueue.QueueStatus{PendingWorkloads: 3}))
 
+		ginkgo.By("Admitting workloads")
+		for _, w := range workloads {
+			w.Spec.Admission = testing.MakeAdmission(clusterQueue.Name).
+				Flavor(corev1.ResourceCPU, flavorOnDemand).Obj()
+			gomega.Expect(k8sClient.Update(ctx, w)).To(gomega.Succeed())
+		}
+		gomega.Eventually(func() kueue.QueueStatus {
+			var updatedQueue kueue.Queue
+			gomega.Expect(k8sClient.Get(ctx, client.ObjectKeyFromObject(queue), &updatedQueue)).To(gomega.Succeed())
+			return updatedQueue.Status
+		}, framework.Timeout, framework.Interval).Should(testing.Equal(kueue.QueueStatus{PendingWorkloads: 0}))
+
 		ginkgo.By("Finishing workloads")
 		for _, w := range workloads {
 			w.Status.Conditions = append(w.Status.Conditions, kueue.QueuedWorkloadCondition{
@@ -93,7 +105,7 @@ var _ = ginkgo.Describe("Queue controller", func() {
 		}
 		gomega.Eventually(func() kueue.QueueStatus {
 			var updatedQueue kueue.Queue
-			gomega.Expect(k8sClient.Get(ctx, client.ObjectKeyFromObject(Queue), &updatedQueue)).To(gomega.Succeed())
+			gomega.Expect(k8sClient.Get(ctx, client.ObjectKeyFromObject(queue), &updatedQueue)).To(gomega.Succeed())
 			return updatedQueue.Status
 		}, framework.Timeout, framework.Interval).Should(testing.Equal(kueue.QueueStatus{}))
 	})