Repository: kubernetes-sigs/karpenter
Pull request #2025: perf: respect multinode consolidation timeout in all cases
Author: login: rschalo
Labels: name: cncf-cla: yes
description: Indicates the PR's author has signed the CNCF CLA.
name: lgtm
description: "Looks good to me", indicates that a PR is ready to be merged.
name: approved
description: Indicates a PR has been approved by an approver from all required OWNERS files.
name: size/L
description: Denotes a PR that changes 100-499 lines, ignoring generated files.
Created: 2025-02-21T21:42:24Z | Merged: 2025-03-19T20:38:33Z

<!-- Please follow the guidelines at https://www.conventionalcommits.org/en/v1.0.0/ and use one of the following in your title:
feat:            <-- New features that require a MINOR version update
fix:             <-- Bug fixes that require at PATCH version update
chore:           <-- Smaller changes that impact behavior but aren't large enough to be features
perf:            <-- Code changes that improve performance but do not impact behavior
docs:            <-- Documentation changes that do not impact code
test:            <-- Test changes that do not impact behavior
ci:              <-- Changes that affect test or rollout automation
!${type}:        <-- Include ! if your change includes a backwards incompatible change.
-->

Fixes #N/A <!-- issue number -->

**Description**
Scheduling for 50 nodes in multinode consolidation can take a long time, especially in large clusters where a scheduling decision for a node can take 20 seconds or longer. This can cause multinode consolidation to block drift, emptiness, and single node consolidation for longer than intended.

**How was this change tested?**
Deployed with a 5 second timeout and saw multinode consolidation bail before exhausting the list of candidates.
```
disruption/multinodeconsolidation.go:83","message":"failed to find a multi-node consolidation after timeout, last considered batch had 8 candidates
```

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.



Diff:
diff --git a/pkg/controllers/disruption/helpers.go b/pkg/controllers/disruption/helpers.go
index ec81a92565..0e59a9e823 100644
--- a/pkg/controllers/disruption/helpers.go
+++ b/pkg/controllers/disruption/helpers.go
@@ -88,7 +88,11 @@ func SimulateScheduling(ctx context.Context, kubeClient client.Client, cluster *
 		return client.ObjectKeyFromObject(p), nil
 	})
 
-	results := scheduler.Solve(log.IntoContext(ctx, operatorlogging.NopLogger), pods).TruncateInstanceTypes(scheduling.MaxInstanceTypes)
+	results, err := scheduler.Solve(log.IntoContext(ctx, operatorlogging.NopLogger), pods)
+	if err != nil {
+		return scheduling.Results{}, fmt.Errorf("scheduling pods, %w", err)
+	}
+	results = results.TruncateInstanceTypes(scheduling.MaxInstanceTypes)
 	for _, n := range results.ExistingNodes {
 		// We consider existing nodes for scheduling. When these nodes are unmanaged, their taint logic should
 		// tell us if we can schedule to them or not; however, if these nodes are managed, we will still schedule to them
diff --git a/pkg/controllers/disruption/multinodeconsolidation.go b/pkg/controllers/disruption/multinodeconsolidation.go
index 58028cf372..c7112260d1 100644
--- a/pkg/controllers/disruption/multinodeconsolidation.go
+++ b/pkg/controllers/disruption/multinodeconsolidation.go
@@ -18,6 +18,7 @@ package disruption
 
 import (
 	"context"
+	"errors"
 	"fmt"
 	"math"
 	"time"
@@ -107,6 +108,7 @@ func (m *MultiNodeConsolidation) ComputeCommand(ctx context.Context, disruptionB
 
 // firstNConsolidationOption looks at the first N NodeClaims to determine if they can all be consolidated at once.  The
 // NodeClaims are sorted by increasing disruption order which correlates to likelihood of being able to consolidate the node
+// nolint:gocyclo
 func (m *MultiNodeConsolidation) firstNConsolidationOption(ctx context.Context, candidates []*Candidate, max int) (Command, scheduling.Results, error) {
 	// we always operate on at least two NodeClaims at once, for single NodeClaims standard consolidation will find all solutions
 	if len(candidates) < 2 {
@@ -120,23 +122,26 @@ func (m *MultiNodeConsolidation) firstNConsolidationOption(ctx context.Context,
 	lastSavedCommand := Command{}
 	lastSavedResults := scheduling.Results{}
 	// Set a timeout
-	timeout := m.clock.Now().Add(MultiNodeConsolidationTimeoutDuration)
-	// binary search to find the maximum number of NodeClaims we can terminate
+	timeoutCtx, cancel := context.WithTimeout(ctx, MultiNodeConsolidationTimeoutDuration)
+	defer cancel()
 	for min <= max {
-		if m.clock.Now().After(timeout) {
-			ConsolidationTimeoutsTotal.Inc(map[string]string{consolidationTypeLabel: m.ConsolidationType()})
-			if lastSavedCommand.candidates == nil {
-				log.FromContext(ctx).V(1).Info(fmt.Sprintf("failed to find a multi-node consolidation after timeout, last considered batch had %d", (min+max)/2))
-			} else {
-				log.FromContext(ctx).V(1).WithValues(lastSavedCommand.LogValues()...).Info("stopping multi-node consolidation after timeout, returning last valid command")
-			}
-			return lastSavedCommand, lastSavedResults, nil
-		}
 		mid := (min + max) / 2
 		candidatesToConsolidate := candidates[0 : mid+1]
 
-		cmd, results, err := m.computeConsolidation(ctx, candidatesToConsolidate...)
+		// Pass the timeout context to ensure sub-operations can be canceled
+		cmd, results, err := m.computeConsolidation(timeoutCtx, candidatesToConsolidate...)
+		// context deadline exceeded will return to the top of the loop and either return nothing or the last saved command
 		if err != nil {
+			if errors.Is(err, context.DeadlineExceeded) {
+				ConsolidationTimeoutsTotal.Inc(map[string]string{consolidationTypeLabel: m.ConsolidationType()})
+				if lastSavedCommand.candidates == nil {
+					log.FromContext(ctx).V(1).Info(fmt.Sprintf("failed to find a multi-node consolidation after timeout, last considered batch had %d", (min+max)/2))
+					return Command{}, scheduling.Results{}, nil
+				}
+				log.FromContext(ctx).V(1).WithValues(lastSavedCommand.LogValues()...).Info("stopping multi-node consolidation after timeout, returning last valid command")
+				return lastSavedCommand, lastSavedResults, nil
+
+			}
 			return Command{}, scheduling.Results{}, err
 		}
 
diff --git a/pkg/controllers/disruption/suite_test.go b/pkg/controllers/disruption/suite_test.go
index a40b7dae0f..26c2e93e54 100644
--- a/pkg/controllers/disruption/suite_test.go
+++ b/pkg/controllers/disruption/suite_test.go
@@ -2022,6 +2022,62 @@ var _ = Describe("Metrics", func() {
 			"consolidation_type": "multi",
 		})
 	})
+	It("should stop multi-node consolidation after context deadline is reached", func() {
+		nodeClaims, nodes = test.NodeClaimsAndNodes(3, v1.NodeClaim{
+			ObjectMeta: metav1.ObjectMeta{
+				Labels: map[string]string{
+					v1.NodePoolLabelKey:            nodePool.Name,
+					corev1.LabelInstanceTypeStable: mostExpensiveInstance.Name,
+					v1.CapacityTypeLabelKey:        mostExpensiveOffering.Requirements.Get(v1.CapacityTypeLabelKey).Any(),
+					corev1.LabelTopologyZone:       mostExpensiveOffering.Requirements.Get(corev1.LabelTopologyZone).Any(),
+				},
+			},
+			Status: v1.NodeClaimStatus{
+				Allocatable: map[corev1.ResourceName]resource.Quantity{
+					corev1.ResourceCPU:  resource.MustParse("32"),
+					corev1.ResourcePods: resource.MustParse("100"),
+				},
+			},
+		})
+		for _, nc := range nodeClaims {
+			nc.StatusConditions().SetTrue(v1.ConditionTypeConsolidatable)
+		}
+		// create our RS so we can link a pod to it
+		rs := test.ReplicaSet()
+		ExpectApplied(ctx, env.Client, rs)
+		pods := test.Pods(4, test.PodOptions{
+			ObjectMeta: metav1.ObjectMeta{Labels: labels,
+				OwnerReferences: []metav1.OwnerReference{
+					{
+						APIVersion:         "apps/v1",
+						Kind:               "ReplicaSet",
+						Name:               rs.Name,
+						UID:                rs.UID,
+						Controller:         lo.ToPtr(true),
+						BlockOwnerDeletion: lo.ToPtr(true),
+					},
+				},
+			},
+		})
+
+		ExpectApplied(ctx, env.Client, rs, pods[0], pods[1], pods[2], pods[3], nodeClaims[0], nodes[0], nodeClaims[1], nodes[1], nodeClaims[2], nodes[2], nodePool)
+
+		// bind pods to nodes
+		ExpectManualBinding(ctx, env.Client, pods[0], nodes[0])
+		ExpectManualBinding(ctx, env.Client, pods[1], nodes[1])
+		ExpectManualBinding(ctx, env.Client, pods[2], nodes[2])
+		ExpectManualBinding(ctx, env.Client, pods[3], nodes[2])
+
+		// inform cluster state about nodes and nodeclaims
+		ExpectMakeNodesAndNodeClaimsInitializedAndStateUpdated(ctx, env.Client, nodeStateController, nodeClaimStateController, []*corev1.Node{nodes[0], nodes[1], nodes[2]}, []*v1.NodeClaim{nodeClaims[0], nodeClaims[1], nodeClaims[2]})
+		// create timeout in the past
+		timeoutCtx, cancel := context.WithTimeout(ctx, -disruption.MultiNodeConsolidationTimeoutDuration)
+		defer cancel()
+
+		ExpectSingletonReconciled(timeoutCtx, disruptionController)
+		// expect that due to timeout zero nodes were tainted in consolidation
+		ExpectTaintedNodeCount(ctx, env.Client, 0)
+	})
 })
 
 func leastExpensiveInstanceWithZone(zone string) *cloudprovider.InstanceType {
diff --git a/pkg/controllers/provisioning/provisioner.go b/pkg/controllers/provisioning/provisioner.go
index 0cbfc1a575..88c12774f2 100644
--- a/pkg/controllers/provisioning/provisioner.go
+++ b/pkg/controllers/provisioning/provisioner.go
@@ -241,9 +241,11 @@ func (p *Provisioner) NewScheduler(
 
 	instanceTypes := map[string][]*cloudprovider.InstanceType{}
 	for _, np := range nodePools {
-		// Get instance type options
 		its, err := p.cloudProvider.GetInstanceTypes(ctx, np)
 		if err != nil {
+			if errors.Is(err, context.DeadlineExceeded) {
+				return nil, fmt.Errorf("getting instance types, %w", err)
+			}
 			log.FromContext(ctx).WithValues("NodePool", klog.KObj(np)).Error(err, "skipping, unable to resolve instance types")
 			continue
 		}
@@ -255,9 +257,12 @@ func (p *Provisioner) NewScheduler(
 	}
 
 	// inject topology constraints
-	pods = p.injectVolumeTopologyRequirements(ctx, pods)
+	pods, err = p.injectVolumeTopologyRequirements(ctx, pods)
+	if err != nil {
+		return nil, fmt.Errorf("injecting volume topology requirements, %w", err)
+	}
 
-	// Calculate cluster topology
+	// Calculate cluster topology, if a context error occurs, it is wrapped and returned
 	topology, err := scheduler.NewTopology(ctx, p.kubeClient, p.cluster, stateNodes, nodePools, instanceTypes, pods)
 	if err != nil {
 		return nil, fmt.Errorf("tracking topology counts, %w", err)
@@ -317,7 +322,12 @@ func (p *Provisioner) Schedule(ctx context.Context) (scheduler.Results, error) {
 		return scheduler.Results{}, fmt.Errorf("creating scheduler, %w", err)
 	}
 
-	results := s.Solve(ctx, pods).TruncateInstanceTypes(scheduler.MaxInstanceTypes)
+	results, err := s.Solve(ctx, pods)
+	// context errors are ignored because we want to finish provisioning for what has already been scheduled
+	if err != nil && !errors.Is(err, context.DeadlineExceeded) {
+		return scheduler.Results{}, err
+	}
+	results = results.TruncateInstanceTypes(scheduler.MaxInstanceTypes)
 	reservedOfferingErrors := results.ReservedOfferingErrors()
 	if len(reservedOfferingErrors) != 0 {
 		log.FromContext(ctx).V(1).WithValues(
@@ -464,16 +474,19 @@ func validateKarpenterManagedLabelCanExist(p *corev1.Pod) error {
 	return nil
 }
 
-func (p *Provisioner) injectVolumeTopologyRequirements(ctx context.Context, pods []*corev1.Pod) []*corev1.Pod {
+func (p *Provisioner) injectVolumeTopologyRequirements(ctx context.Context, pods []*corev1.Pod) ([]*corev1.Pod, error) {
 	var schedulablePods []*corev1.Pod
 	for _, pod := range pods {
 		if err := p.volumeTopology.Inject(ctx, pod); err != nil {
+			if errors.Is(err, context.DeadlineExceeded) {
+				return nil, err
+			}
 			log.FromContext(ctx).WithValues("Pod", klog.KObj(pod)).Error(err, "failed getting volume topology requirements")
 		} else {
 			schedulablePods = append(schedulablePods, pod)
 		}
 	}
-	return schedulablePods
+	return schedulablePods, nil
 }
 
 func validateNodeSelector(p *corev1.Pod) (errs error) {
diff --git a/pkg/controllers/provisioning/scheduling/scheduler.go b/pkg/controllers/provisioning/scheduling/scheduler.go
index 152998c50f..8b6c74f9c9 100644
--- a/pkg/controllers/provisioning/scheduling/scheduler.go
+++ b/pkg/controllers/provisioning/scheduling/scheduler.go
@@ -19,6 +19,7 @@ package scheduling
 import (
 	"bytes"
 	"context"
+	"errors"
 	"fmt"
 	"sort"
 	"time"
@@ -266,14 +267,14 @@ func (r Results) TruncateInstanceTypes(maxInstanceTypes int) Results {
 }
 
 //nolint:gocyclo
-func (s *Scheduler) Solve(ctx context.Context, pods []*corev1.Pod) Results {
+func (s *Scheduler) Solve(ctx context.Context, pods []*corev1.Pod) (Results, error) {
 	defer metrics.Measure(DurationSeconds, map[string]string{ControllerLabel: injection.GetControllerName(ctx)})()
 	// We loop trying to schedule unschedulable pods as long as we are making progress.  This solves a few
 	// issues including pods with affinity to another pod in the batch. We could topo-sort to solve this, but it wouldn't
 	// solve the problem of scheduling pods where a particular order is needed to prevent a max-skew violation. E.g. if we
 	// had 5xA pods and 5xB pods were they have a zonal topology spread, but A can only go in one zone and B in another.
 	// We need to schedule them alternating, A, B, A, B, .... and this solution also solves that as well.
-	errors := map[*corev1.Pod]error{}
+	podErrors := map[*corev1.Pod]error{}
 	// Reset the metric for the controller, so we don't keep old ids around
 	UnschedulablePodsCount.DeletePartialMatch(map[string]string{ControllerLabel: injection.GetControllerName(ctx)})
 	QueueDepth.DeletePartialMatch(map[string]string{ControllerLabel: injection.GetControllerName(ctx)})
@@ -286,6 +287,10 @@ func (s *Scheduler) Solve(ctx context.Context, pods []*corev1.Pod) Results {
 	lastLogTime := s.clock.Now()
 	batchSize := len(q.pods)
 	for {
+		if ctx.Err() != nil {
+			log.FromContext(ctx).V(1).WithValues("duration", s.clock.Since(startTime).Truncate(time.Second), "scheduling-id", string(s.uuid)).Info("scheduling simulation timed out")
+			break
+		}
 		UnfinishedWorkSeconds.Set(s.clock.Since(startTime).Seconds(), map[string]string{ControllerLabel: injection.GetControllerName(ctx), schedulingIDLabel: string(s.uuid)})
 		QueueDepth.Set(float64(len(q.pods)), map[string]string{ControllerLabel: injection.GetControllerName(ctx), schedulingIDLabel: string(s.uuid)})
 
@@ -300,8 +305,8 @@ func (s *Scheduler) Solve(ctx context.Context, pods []*corev1.Pod) Results {
 		}
 
 		// Schedule to existing nodes or create a new node
-		if errors[pod] = s.add(ctx, pod); errors[pod] == nil {
-			delete(errors, pod)
+		if podErrors[pod] = s.add(ctx, pod); podErrors[pod] == nil {
+			delete(podErrors, pod)
 			continue
 		}
 
@@ -310,9 +315,9 @@ func (s *Scheduler) Solve(ctx context.Context, pods []*corev1.Pod) Results {
 		// release the required reservations when constrained, or in subsequent runs. For an example, reference the following
 		// test: "shouldn't relax preferences when a pod fails to schedule due to a reserved offering error".
 		var relaxed bool
-		if !IsReservedOfferingError(errors[pod]) {
+		if !IsReservedOfferingError(podErrors[pod]) {
 			if relaxed = s.preferences.Relax(ctx, pod); relaxed {
-				if err := s.topology.Update(ctx, pod); err != nil {
+				if err := s.topology.Update(ctx, pod); err != nil && !errors.Is(err, context.DeadlineExceeded) {
 					log.FromContext(ctx).Error(err, "failed updating topology")
 				}
 				// Update the cached podData since the pod was relaxed and it could have changed its requirement set
@@ -329,8 +334,8 @@ func (s *Scheduler) Solve(ctx context.Context, pods []*corev1.Pod) Results {
 	return Results{
 		NewNodeClaims: s.newNodeClaims,
 		ExistingNodes: s.existingNodes,
-		PodErrors:     errors,
-	}
+		PodErrors:     podErrors,
+	}, ctx.Err()
 }
 
 func (s *Scheduler) updateCachedPodData(p *corev1.Pod) {
diff --git a/pkg/controllers/provisioning/scheduling/scheduling_benchmark_test.go b/pkg/controllers/provisioning/scheduling/scheduling_benchmark_test.go
index 3907ca5242..b03a7bc0ad 100644
--- a/pkg/controllers/provisioning/scheduling/scheduling_benchmark_test.go
+++ b/pkg/controllers/provisioning/scheduling/scheduling_benchmark_test.go
@@ -182,7 +182,10 @@ func benchmarkScheduler(b *testing.B, instanceCount, podCount int) {
 	podsScheduledInRound1 := 0
 	nodesInRound1 := 0
 	for i := 0; i < b.N; i++ {
-		results := scheduler.Solve(ctx, pods)
+		results, err := scheduler.Solve(ctx, pods)
+		if err != nil {
+			b.Fatalf("expected scheduler to schedule all pods without error, got %s", err)
+		}
 		if len(results.PodErrors) > 0 {
 			b.Fatalf("expected all pods to schedule, got %d pods that didn't", len(results.PodErrors))
 		}
diff --git a/pkg/controllers/provisioning/scheduling/suite_test.go b/pkg/controllers/provisioning/scheduling/suite_test.go
index e89c723d7d..90e7c19df2 100644
--- a/pkg/controllers/provisioning/scheduling/suite_test.go
+++ b/pkg/controllers/provisioning/scheduling/suite_test.go
@@ -3682,7 +3682,9 @@ var _ = Context("Scheduling", func() {
 					g.Expect(lo.FromPtr(m.Gauge.Value)).To(BeNumerically(">", 0))
 				}, time.Second).Should(Succeed())
 			}()
-			s.Solve(injection.WithControllerName(ctx, "provisioner"), pods)
+			_, err = s.Solve(injection.WithControllerName(ctx, "provisioner"), pods)
+			Expect(err).To(BeNil())
+
 			wg.Wait()
 		})
 		It("should surface the UnschedulablePodsCount metric while executing the scheduling loop", func() {
@@ -3742,7 +3744,8 @@ var _ = Context("Scheduling", func() {
 			}) // Create 1000 pods which should take long enough to schedule that we should be able to read the queueDepth metric with a value
 			s, err := prov.NewScheduler(ctx, pods, nil, scheduling.DisableReservedCapacityFallback)
 			Expect(err).To(BeNil())
-			s.Solve(injection.WithControllerName(ctx, "provisioner"), pods)
+			_, err = s.Solve(injection.WithControllerName(ctx, "provisioner"), pods)
+			Expect(err).To(BeNil())
 
 			m, ok := FindMetricWithLabelValues("karpenter_scheduler_scheduling_duration_seconds", map[string]string{"controller": "provisioner"})
 			Expect(ok).To(BeTrue())


Reviews:
user: login: tzneal
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-21T22:03:26Z
user: login: rschalo
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-24T23:52:20Z
user: login: tzneal
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-25T21:35:47Z
user: login: rschalo
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-02-25T22:55:24Z
user: login: jonathan-innis
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-04T21:36:58Z
user: login: rschalo
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-04T23:52:50Z
user: login: rschalo
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-04T23:57:20Z
user: login: jonathan-innis
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-07T08:02:42Z
user: login: rschalo
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-07T23:21:24Z
user: login: rschalo
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-07T23:46:19Z
user: login: jonathan-innis
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-17T05:22:32Z
user: login: jonathan-innis
body: /lgtm
/approve
state: APPROVED
author_association: MEMBER
submitted_at: 2025-03-19T20:25:55Z

Review comments:
diff_hunk: @@ -120,42 +120,48 @@ func (m *MultiNodeConsolidation) firstNConsolidationOption(ctx context.Context,
 	lastSavedCommand := Command{}
 	lastSavedResults := scheduling.Results{}
 	// Set a timeout
-	timeout := m.clock.Now().Add(MultiNodeConsolidationTimeoutDuration)
-	// binary search to find the maximum number of NodeClaims we can terminate
+	timeoutCtx, cancel := context.WithDeadline(ctx, m.clock.Now().Add(MultiNodeConsolidationTimeoutDuration))
+	defer cancel()
 	for min <= max {
-		if m.clock.Now().After(timeout) {
+		// Check for timeout using select
+		select {
+		case <-timeoutCtx.Done():
 			ConsolidationTimeoutsTotal.Inc(map[string]string{consolidationTypeLabel: m.ConsolidationType()})
 			if lastSavedCommand.candidates == nil {
-				log.FromContext(ctx).V(1).Info(fmt.Sprintf("failed to find a multi-node consolidation after timeout, last considered batch had %d", (min+max)/2))
+				log.FromContext(ctx).V(1).Info(fmt.Sprintf("failed to find a multi-node consolidation after timeout, last considered batch had %d nodes", (min+max)/2))
 			} else {
 				log.FromContext(ctx).V(1).Info(fmt.Sprintf("stopping multi-node consolidation after timeout, returning last valid command %s", lastSavedCommand))
 			}
 			return lastSavedCommand, lastSavedResults, nil
-		}
-		mid := (min + max) / 2
-		candidatesToConsolidate := candidates[0 : mid+1]
+		default:
+			mid := (min + max) / 2
+			candidatesToConsolidate := candidates[0 : mid+1]
+
+			// Pass the timeout context to ensure sub-operations can be canceled
+			cmd, results, err := m.computeConsolidation(timeoutCtx, candidatesToConsolidate...)
+			if err != nil {
+				return Command{}, scheduling.Results{}, err
+			}
 
-		cmd, results, err := m.computeConsolidation(ctx, candidatesToConsolidate...)
-		if err != nil {
-			return Command{}, scheduling.Results{}, err
-		}
+			// ensure that the action is sensical for replacements, see explanation on filterOutSameType for why this is
+			// required
+			replacementHasValidInstanceTypes := false
+			if cmd.Decision() == ReplaceDecision {
+				log.FromContext(ctx).V(1).Info(fmt.Sprintf("evaluating replace decision, candidates: %v, replacements: %v", len(lastSavedCommand.candidates), len(lastSavedCommand.replacements)))
path: pkg/controllers/disruption/multinodeconsolidation.go
user: login: tzneal
body: Do we need this log?
created_at: 2025-02-21T22:01:35Z
author_association: CONTRIBUTOR
diff_hunk: @@ -120,42 +120,48 @@ func (m *MultiNodeConsolidation) firstNConsolidationOption(ctx context.Context,
 	lastSavedCommand := Command{}
 	lastSavedResults := scheduling.Results{}
 	// Set a timeout
-	timeout := m.clock.Now().Add(MultiNodeConsolidationTimeoutDuration)
-	// binary search to find the maximum number of NodeClaims we can terminate
+	timeoutCtx, cancel := context.WithDeadline(ctx, m.clock.Now().Add(MultiNodeConsolidationTimeoutDuration))
+	defer cancel()
 	for min <= max {
-		if m.clock.Now().After(timeout) {
+		// Check for timeout using select
+		select {
+		case <-timeoutCtx.Done():
 			ConsolidationTimeoutsTotal.Inc(map[string]string{consolidationTypeLabel: m.ConsolidationType()})
 			if lastSavedCommand.candidates == nil {
-				log.FromContext(ctx).V(1).Info(fmt.Sprintf("failed to find a multi-node consolidation after timeout, last considered batch had %d", (min+max)/2))
+				log.FromContext(ctx).V(1).Info(fmt.Sprintf("failed to find a multi-node consolidation after timeout, last considered batch had %d nodes", (min+max)/2))
 			} else {
 				log.FromContext(ctx).V(1).Info(fmt.Sprintf("stopping multi-node consolidation after timeout, returning last valid command %s", lastSavedCommand))
 			}
 			return lastSavedCommand, lastSavedResults, nil
-		}
-		mid := (min + max) / 2
-		candidatesToConsolidate := candidates[0 : mid+1]
+		default:
+			mid := (min + max) / 2
+			candidatesToConsolidate := candidates[0 : mid+1]
+
+			// Pass the timeout context to ensure sub-operations can be canceled
+			cmd, results, err := m.computeConsolidation(timeoutCtx, candidatesToConsolidate...)
path: pkg/controllers/disruption/multinodeconsolidation.go
user: login: tzneal
body: Won't the context timeout produce an err here that we need to handle specially?

I was also expecting we might need to check for context cancellation in the scheduler more, but maybe its getting caught when we do something like list pods/nodes?
created_at: 2025-02-21T22:02:04Z
author_association: CONTRIBUTOR
diff_hunk: @@ -120,42 +120,48 @@ func (m *MultiNodeConsolidation) firstNConsolidationOption(ctx context.Context,
 	lastSavedCommand := Command{}
 	lastSavedResults := scheduling.Results{}
 	// Set a timeout
-	timeout := m.clock.Now().Add(MultiNodeConsolidationTimeoutDuration)
-	// binary search to find the maximum number of NodeClaims we can terminate
+	timeoutCtx, cancel := context.WithDeadline(ctx, m.clock.Now().Add(MultiNodeConsolidationTimeoutDuration))
+	defer cancel()
 	for min <= max {
-		if m.clock.Now().After(timeout) {
+		// Check for timeout using select
+		select {
+		case <-timeoutCtx.Done():
 			ConsolidationTimeoutsTotal.Inc(map[string]string{consolidationTypeLabel: m.ConsolidationType()})
 			if lastSavedCommand.candidates == nil {
-				log.FromContext(ctx).V(1).Info(fmt.Sprintf("failed to find a multi-node consolidation after timeout, last considered batch had %d", (min+max)/2))
+				log.FromContext(ctx).V(1).Info(fmt.Sprintf("failed to find a multi-node consolidation after timeout, last considered batch had %d nodes", (min+max)/2))
 			} else {
 				log.FromContext(ctx).V(1).Info(fmt.Sprintf("stopping multi-node consolidation after timeout, returning last valid command %s", lastSavedCommand))
 			}
 			return lastSavedCommand, lastSavedResults, nil
-		}
-		mid := (min + max) / 2
-		candidatesToConsolidate := candidates[0 : mid+1]
+		default:
+			mid := (min + max) / 2
+			candidatesToConsolidate := candidates[0 : mid+1]
+
+			// Pass the timeout context to ensure sub-operations can be canceled
+			cmd, results, err := m.computeConsolidation(timeoutCtx, candidatesToConsolidate...)
path: pkg/controllers/disruption/multinodeconsolidation.go
user: login: rschalo
body: So, when I looked we wrap this error in many cases. Additionally, there is a no-op logger for the scheduler that is created for consolidation actions. The trickiest part has been deciding where we want to make cutoffs, for example in `Solve` on `pkg/controllers/provisioning/scheduling/scheduler.go:214`.

The stance I've taken is that we'll allow the current item being evaluated to finish or error out with a wrapped error and since errors are aggregated, break out of the loop and report the next pod as unschedulable.
created_at: 2025-02-24T23:52:20Z
author_association: CONTRIBUTOR
diff_hunk: @@ -120,42 +122,50 @@ func (m *MultiNodeConsolidation) firstNConsolidationOption(ctx context.Context,
 	lastSavedCommand := Command{}
 	lastSavedResults := scheduling.Results{}
 	// Set a timeout
-	timeout := m.clock.Now().Add(MultiNodeConsolidationTimeoutDuration)
-	// binary search to find the maximum number of NodeClaims we can terminate
+	timeoutCtx, cancel := context.WithDeadline(ctx, m.clock.Now().Add(MultiNodeConsolidationTimeoutDuration))
+	defer cancel()
 	for min <= max {
-		if m.clock.Now().After(timeout) {
+		// Check for timeout using select
+		select {
path: pkg/controllers/disruption/multinodeconsolidation.go
user: login: tzneal
body: Can we just check for `if ctx.Err() != nil` here, and does that make this a bit cleaner? (no select effectively and a smaller diff)
created_at: 2025-02-25T21:30:48Z
author_association: CONTRIBUTOR
diff_hunk: @@ -106,6 +107,10 @@ func SimulateScheduling(ctx context.Context, kubeClient client.Client, cluster *
 				if _, ok := deletingNodePodKeys[client.ObjectKeyFromObject(p)]; !ok {
 					results.PodErrors[p] = NewUninitializedNodeError(n)
 				}
+				// check for a pod that was not scheduled due to ctx timeout
path: pkg/controllers/disruption/helpers.go
user: login: tzneal
body: I'm probably missing something, but what is this for? Shouldn't we normally fail scheduling if there is a pod error?
created_at: 2025-02-25T21:35:15Z
author_association: CONTRIBUTOR
diff_hunk: @@ -106,6 +107,10 @@ func SimulateScheduling(ctx context.Context, kubeClient client.Client, cluster *
 				if _, ok := deletingNodePodKeys[client.ObjectKeyFromObject(p)]; !ok {
 					results.PodErrors[p] = NewUninitializedNodeError(n)
 				}
+				// check for a pod that was not scheduled due to ctx timeout
path: pkg/controllers/disruption/helpers.go
user: login: rschalo
body: This is probably unnecessary. Was using it to error on line 142 after calling `SimulateScheduling` in `computeConsolidation` but we do check if pods failed to schedule on line 146. We'd then return empty results, no command nil error, and continue the loop. 

The loop then either breaks because min > max and returns the last saved items or the timeout bails out of consolidation and returns the last saved items plus an error if context deadline was exceeded.
created_at: 2025-02-25T22:55:24Z
author_association: CONTRIBUTOR
diff_hunk: @@ -120,22 +122,27 @@ func (m *MultiNodeConsolidation) firstNConsolidationOption(ctx context.Context,
 	lastSavedCommand := Command{}
 	lastSavedResults := scheduling.Results{}
 	// Set a timeout
-	timeout := m.clock.Now().Add(MultiNodeConsolidationTimeoutDuration)
-	// binary search to find the maximum number of NodeClaims we can terminate
+	timeoutCtx, cancel := context.WithDeadline(ctx, m.clock.Now().Add(MultiNodeConsolidationTimeoutDuration))
path: pkg/controllers/disruption/multinodeconsolidation.go
user: login: jonathan-innis
body: ```suggestion
	timeoutCtx, cancel := context.WithTimeout(ctx, MultiNodeConsolidationTimeoutDuration)
```

nit: A bit less verbose
created_at: 2025-03-04T21:21:50Z
author_association: MEMBER
diff_hunk: @@ -120,22 +122,27 @@ func (m *MultiNodeConsolidation) firstNConsolidationOption(ctx context.Context,
 	lastSavedCommand := Command{}
 	lastSavedResults := scheduling.Results{}
 	// Set a timeout
-	timeout := m.clock.Now().Add(MultiNodeConsolidationTimeoutDuration)
-	// binary search to find the maximum number of NodeClaims we can terminate
+	timeoutCtx, cancel := context.WithDeadline(ctx, m.clock.Now().Add(MultiNodeConsolidationTimeoutDuration))
+	defer cancel()
 	for min <= max {
-		if m.clock.Now().After(timeout) {
+		// Check for timeout using select
+		if timeoutCtx.Err() != nil {
 			ConsolidationTimeoutsTotal.Inc(map[string]string{consolidationTypeLabel: m.ConsolidationType()})
 			if lastSavedCommand.candidates == nil {
-				log.FromContext(ctx).V(1).Info(fmt.Sprintf("failed to find a multi-node consolidation after timeout, last considered batch had %d", (min+max)/2))
-			} else {
-				log.FromContext(ctx).V(1).WithValues(lastSavedCommand.LogValues()...).Info("stopping multi-node consolidation after timeout, returning last valid command")
+				return Command{}, scheduling.Results{}, fmt.Errorf("multi-node consolidation timed out while considering %d nodes without finding a valid command", (min+max)/2)
 			}
+			log.FromContext(ctx).V(1).WithValues(lastSavedCommand.LogValues()...).Info("stopping multi-node consolidation after timeout, returning last valid command")
 			return lastSavedCommand, lastSavedResults, nil
 		}
 		mid := (min + max) / 2
 		candidatesToConsolidate := candidates[0 : mid+1]
 
-		cmd, results, err := m.computeConsolidation(ctx, candidatesToConsolidate...)
+		// Pass the timeout context to ensure sub-operations can be canceled
+		cmd, results, err := m.computeConsolidation(timeoutCtx, candidatesToConsolidate...)
+		// context deadline exceeded will return to the top of the loop and either return nothing or the last saved command
+		if errors.Is(err, context.DeadlineExceeded) {
path: pkg/controllers/disruption/multinodeconsolidation.go
user: login: jonathan-innis
body: nit: Checking this inside of the `err != nil` check is the typical pattern
created_at: 2025-03-04T21:25:24Z
author_association: MEMBER
diff_hunk: @@ -244,6 +244,12 @@ func (s *Scheduler) Solve(ctx context.Context, pods []*corev1.Pod) Results {
 			break
 		}
 
+		// when context has been canceled or deadline exceeded, stop attempting to schedule pods and mark current pod as unschedulable
+		if ctx.Err() != nil {
+			errors[pod] = ctx.Err()
+			break
path: pkg/controllers/provisioning/scheduling/scheduler.go
user: login: jonathan-innis
body: I'm curious what you think about this in the context of applying a scheduling timeout as well -- we can choose to either: 1) Mark all pods that have not been scheduled yet as failing due to context id -- this could take some time potentially depending on the number of pods that we have OR 2) Fail the scheduling loop with a context timeout error -- in that case, we'd have to somehow know that we didn't schedule all the pods -- given that we are using errors to communicate this right now, that might not work well
created_at: 2025-03-04T21:33:32Z
author_association: MEMBER
diff_hunk: @@ -244,6 +244,12 @@ func (s *Scheduler) Solve(ctx context.Context, pods []*corev1.Pod) Results {
 			break
 		}
 
+		// when context has been canceled or deadline exceeded, stop attempting to schedule pods and mark current pod as unschedulable
+		if ctx.Err() != nil {
+			errors[pod] = ctx.Err()
+			break
path: pkg/controllers/provisioning/scheduling/scheduler.go
user: login: jonathan-innis
body: In either case, I don't think we can break here because I think we need to mark all pods that weren't scheduled as unschedulable to have a proper interface
created_at: 2025-03-04T21:34:25Z
author_association: MEMBER
diff_hunk: @@ -236,6 +236,9 @@ func (p *Provisioner) NewScheduler(ctx context.Context, pods []*corev1.Pod, stat
 
 	instanceTypes := map[string][]*cloudprovider.InstanceType{}
 	for _, np := range nodePools {
+		if ctx.Err() != nil {
path: pkg/controllers/provisioning/provisioner.go
user: login: jonathan-innis
body: Does this section of code take so much time that we think we need to handle this error at this level? I get that there's a trade-off between the number of times that we write this and how quickly we can respond 
created_at: 2025-03-04T21:36:04Z
author_association: MEMBER
diff_hunk: @@ -2022,6 +2022,62 @@ var _ = Describe("Metrics", func() {
 			"consolidation_type": "multi",
 		})
 	})
+	It("should stop multi-node consolidation after context deadline is reached", func() {
+		nodeClaims, nodes = test.NodeClaimsAndNodes(3, v1.NodeClaim{
+			ObjectMeta: metav1.ObjectMeta{
+				Labels: map[string]string{
+					v1.NodePoolLabelKey:            nodePool.Name,
+					corev1.LabelInstanceTypeStable: mostExpensiveInstance.Name,
+					v1.CapacityTypeLabelKey:        mostExpensiveOffering.Requirements.Get(v1.CapacityTypeLabelKey).Any(),
+					corev1.LabelTopologyZone:       mostExpensiveOffering.Requirements.Get(corev1.LabelTopologyZone).Any(),
+				},
+			},
+			Status: v1.NodeClaimStatus{
+				Allocatable: map[corev1.ResourceName]resource.Quantity{
+					corev1.ResourceCPU:  resource.MustParse("32"),
+					corev1.ResourcePods: resource.MustParse("100"),
+				},
+			},
+		})
+		for _, nc := range nodeClaims {
+			nc.StatusConditions().SetTrue(v1.ConditionTypeConsolidatable)
+		}
+		// create our RS so we can link a pod to it
+		rs := test.ReplicaSet()
+		ExpectApplied(ctx, env.Client, rs)
+		pods := test.Pods(4, test.PodOptions{
+			ObjectMeta: metav1.ObjectMeta{Labels: labels,
+				OwnerReferences: []metav1.OwnerReference{
+					{
+						APIVersion:         "apps/v1",
+						Kind:               "ReplicaSet",
+						Name:               rs.Name,
+						UID:                rs.UID,
+						Controller:         lo.ToPtr(true),
+						BlockOwnerDeletion: lo.ToPtr(true),
+					},
+				},
+			},
+		})
+
+		ExpectApplied(ctx, env.Client, rs, pods[0], pods[1], pods[2], pods[3], nodeClaims[0], nodes[0], nodeClaims[1], nodes[1], nodeClaims[2], nodes[2], nodePool)
+
+		// bind pods to nodes
+		ExpectManualBinding(ctx, env.Client, pods[0], nodes[0])
+		ExpectManualBinding(ctx, env.Client, pods[1], nodes[1])
+		ExpectManualBinding(ctx, env.Client, pods[2], nodes[2])
+		ExpectManualBinding(ctx, env.Client, pods[3], nodes[2])
+
+		// inform cluster state about nodes and nodeclaims
+		ExpectMakeNodesAndNodeClaimsInitializedAndStateUpdated(ctx, env.Client, nodeStateController, nodeClaimStateController, []*corev1.Node{nodes[0], nodes[1], nodes[2]}, []*v1.NodeClaim{nodeClaims[0], nodeClaims[1], nodeClaims[2]})
+		// create deadline in the past
+		deadlineCtx, cancel := context.WithDeadline(ctx, fakeClock.Now().Add(-disruption.MultiNodeConsolidationTimeoutDuration))
path: pkg/controllers/disruption/suite_test.go
user: login: jonathan-innis
body: I'm not sure if you can create a timeout that's negative but you could consider doing that here
created_at: 2025-03-04T21:36:42Z
author_association: MEMBER
diff_hunk: @@ -236,6 +236,9 @@ func (p *Provisioner) NewScheduler(ctx context.Context, pods []*corev1.Pod, stat
 
 	instanceTypes := map[string][]*cloudprovider.InstanceType{}
 	for _, np := range nodePools {
+		if ctx.Err() != nil {
path: pkg/controllers/provisioning/provisioner.go
user: login: rschalo
body: We probably don't need to check here. It probably makes the most sense to just check the timeout between pods in scheduling.
created_at: 2025-03-04T23:52:50Z
author_association: CONTRIBUTOR
diff_hunk: @@ -236,6 +236,9 @@ func (p *Provisioner) NewScheduler(ctx context.Context, pods []*corev1.Pod, stat
 
 	instanceTypes := map[string][]*cloudprovider.InstanceType{}
 	for _, np := range nodePools {
+		if ctx.Err() != nil {
path: pkg/controllers/provisioning/provisioner.go
user: login: rschalo
body: Except that this context is timed out and continues on to `cloudProvider.GetInstanceTypes`. Less that we're handling the error and more that we're silencing spurious logging.
created_at: 2025-03-04T23:57:20Z
author_association: CONTRIBUTOR
diff_hunk: @@ -326,9 +330,17 @@ func (p *Provisioner) Schedule(ctx context.Context) (scheduler.Results, error) {
 			}), 5),
 		).Info("deferring scheduling decision for provisionable pod(s) to future simulation due to limited reserved offering capacity")
 	}
+	contextErrors := results.ContextDeadlineExceededErrors()
+	if len(contextErrors) != 0 {
+		log.FromContext(ctx).V(1).WithValues(
path: pkg/controllers/provisioning/provisioner.go
user: login: jonathan-innis
body: This along with the log above (for reserved offerings) strike me as debug logs
created_at: 2025-03-07T07:59:03Z
author_association: MEMBER
diff_hunk: @@ -236,6 +236,9 @@ func (p *Provisioner) NewScheduler(ctx context.Context, pods []*corev1.Pod, stat
 
 	instanceTypes := map[string][]*cloudprovider.InstanceType{}
 	for _, np := range nodePools {
+		if ctx.Err() != nil {
path: pkg/controllers/provisioning/provisioner.go
user: login: jonathan-innis
body: That doesn't completely solve it right? I think we just have to handle it generally because we can still race pass this check and fire spurious errors
created_at: 2025-03-07T08:00:19Z
author_association: MEMBER
diff_hunk: @@ -326,9 +330,17 @@ func (p *Provisioner) Schedule(ctx context.Context) (scheduler.Results, error) {
 			}), 5),
 		).Info("deferring scheduling decision for provisionable pod(s) to future simulation due to limited reserved offering capacity")
 	}
+	contextErrors := results.ContextDeadlineExceededErrors()
+	if len(contextErrors) != 0 {
+		log.FromContext(ctx).V(1).WithValues(
path: pkg/controllers/provisioning/provisioner.go
user: login: rschalo
body: Is that not the case with the `.V(1)`?
created_at: 2025-03-07T23:21:24Z
author_association: CONTRIBUTOR
diff_hunk: @@ -326,9 +330,17 @@ func (p *Provisioner) Schedule(ctx context.Context) (scheduler.Results, error) {
 			}), 5),
 		).Info("deferring scheduling decision for provisionable pod(s) to future simulation due to limited reserved offering capacity")
 	}
+	contextErrors := results.ContextDeadlineExceededErrors()
+	if len(contextErrors) != 0 {
+		log.FromContext(ctx).V(1).WithValues(
path: pkg/controllers/provisioning/provisioner.go
user: login: rschalo
body: This is actually not needed at all in this PR. It's something we'll have to figure out if we want to add a timeout to the provisioning path.
created_at: 2025-03-07T23:46:19Z
author_association: CONTRIBUTOR
diff_hunk: @@ -92,7 +97,7 @@ func (d *Drift) ComputeCommand(ctx context.Context, disruptionBudgetMapping map[
 			continue
 		}
 		// Check if we need to create any NodeClaims.
-		results, err := SimulateScheduling(ctx, d.kubeClient, d.cluster, d.provisioner, candidate)
+		results, err := SimulateScheduling(ctx, DriftTimeout, d.kubeClient, d.cluster, d.provisioner, candidate)
path: pkg/controllers/disruption/drift.go
user: login: jonathan-innis
body: Have we actually seen drift taking way too long? Is this for the case where there are pending pods and we just take way to long to simulate? Looks like we are counting this as an error -- which I think probably makes sense because we wouldn't expect this and we'd definitely want to bail-out to re-grab the pending pods, candidates, etc.
created_at: 2025-03-14T00:13:35Z
author_association: MEMBER
diff_hunk: @@ -464,16 +466,20 @@ func validateKarpenterManagedLabelCanExist(p *corev1.Pod) error {
 	return nil
 }
 
-func (p *Provisioner) injectVolumeTopologyRequirements(ctx context.Context, pods []*corev1.Pod) []*corev1.Pod {
+func (p *Provisioner) injectVolumeTopologyRequirements(ctx context.Context, pods []*corev1.Pod) ([]*corev1.Pod, error) {
 	var schedulablePods []*corev1.Pod
 	for _, pod := range pods {
 		if err := p.volumeTopology.Inject(ctx, pod); err != nil {
+			// return because we can't consider any other pod schedulable
path: pkg/controllers/provisioning/provisioner.go
user: login: jonathan-innis
body: What's the reason that we need to handle this here?
created_at: 2025-03-17T05:21:31Z
author_association: MEMBER
diff_hunk: @@ -286,6 +286,10 @@ func (s *Scheduler) Solve(ctx context.Context, pods []*corev1.Pod) Results {
 	lastLogTime := s.clock.Now()
 	batchSize := len(q.pods)
 	for {
+		if ctx.Err() != nil {
+			log.FromContext(ctx).V(1).WithValues("duration", s.clock.Since(startTime).Truncate(time.Second), "scheduling-id", string(s.uuid)).Info("scheduling simulation timed out")
+			break
path: pkg/controllers/provisioning/scheduling/scheduler.go
user: login: jonathan-innis
body: We can't break, right? This means that we won't throw an error back when from the scheduling loop, right?
created_at: 2025-03-17T05:22:23Z
author_association: MEMBER

PR comments:
user: login: coveralls
created_at: 2025-02-21T22:04:46Z
body: ## Pull Request Test Coverage Report for [Build 13956030074](https://coveralls.io/builds/72820813)

### Warning: This coverage report may be inaccurate.

This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

- For more information on this, see <a target="_blank" href="https://docs.coveralls.io/build-types#tracking-coverage-changes-with-pull_request-builds">Tracking coverage changes with pull request builds</a>.
- To avoid this issue with future PRs, see these <a target="_blank" href="https://docs.coveralls.io/build-types#recommended-ci-configurations">Recommended CI Configurations</a>.
- For a quick fix, <a target="_blank" href="https://github.blog/changelog/2022-02-03-more-ways-to-keep-your-pull-request-branch-up-to-date/#update-your-pull-request-branch-by-rebasing">rebase this PR at GitHub</a>. Your next report should be accurate.

### Details

* **25** of **47**   **(53.19%)**  changed or added relevant lines in **4** files are covered.
* **12** unchanged lines in **4** files lost coverage.
* Overall coverage increased (+**0.1%**) to **81.707%**

---

|  Changes Missing Coverage | Covered Lines | Changed/Added Lines | % |
| :-----|--------------|--------|---: |
| [pkg/controllers/disruption/helpers.go](https://coveralls.io/builds/72820813/source?filename=pkg%2Fcontrollers%2Fdisruption%2Fhelpers.go#L93) | 3 | 5 | 60.0%
| [pkg/controllers/provisioning/scheduling/scheduler.go](https://coveralls.io/builds/72820813/source?filename=pkg%2Fcontrollers%2Fprovisioning%2Fscheduling%2Fscheduler.go#L291) | 9 | 11 | 81.82%
| [pkg/controllers/disruption/multinodeconsolidation.go](https://coveralls.io/builds/72820813/source?filename=pkg%2Fcontrollers%2Fdisruption%2Fmultinodeconsolidation.go#L135) | 5 | 13 | 38.46%
| [pkg/controllers/provisioning/provisioner.go](https://coveralls.io/builds/72820813/source?filename=pkg%2Fcontrollers%2Fprovisioning%2Fprovisioner.go#L246) | 8 | 18 | 44.44%
<!-- | **Total:** | **25** | **47** | **53.19%** | -->

|  Files with Coverage Reduction | New Missed Lines | % |
| :-----|--------------|--: |
| [pkg/controllers/provisioning/provisioner.go](https://coveralls.io/builds/72820813/source?filename=pkg%2Fcontrollers%2Fprovisioning%2Fprovisioner.go#L484) | 1 | 83.33% |
| [pkg/test/expectations/expectations.go](https://coveralls.io/builds/72820813/source?filename=pkg%2Ftest%2Fexpectations%2Fexpectations.go#L664) | 2 | 95.0% |
| [pkg/controllers/disruption/consolidation.go](https://coveralls.io/builds/72820813/source?filename=pkg%2Fcontrollers%2Fdisruption%2Fconsolidation.go#L258) | 4 | 85.55% |
| [pkg/controllers/disruption/singlenodeconsolidation.go](https://coveralls.io/builds/72820813/source?filename=pkg%2Fcontrollers%2Fdisruption%2Fsinglenodeconsolidation.go#L55) | 5 | 93.75% |
<!-- | **Total:** | **12** |  | -->

|  Totals | [![Coverage Status](https://coveralls.io/builds/72820813/badge)](https://coveralls.io/builds/72820813) |
| :-- | --: |
| Change from base [Build 13728019186](https://coveralls.io/builds/72602858): |  0.1% |
| Covered Lines: | 9657 |
| Relevant Lines: | 11819 |

---
##### 💛  - [Coveralls](https://coveralls.io)

author_association: NONE
user: login: k8s-ci-robot
created_at: 2025-03-19T20:26:02Z
body: [APPROVALNOTIFIER] This PR is **APPROVED**

This pull-request has been approved by: *<a href="https://github.com/kubernetes-sigs/karpenter/pull/2025#pullrequestreview-2700106170" title="Approved">jonathan-innis</a>*, *<a href="https://github.com/kubernetes-sigs/karpenter/pull/2025#" title="Author self-approved">rschalo</a>*

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=kubernetes-sigs%2Fkarpenter).

The pull request process is described [here](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process)

<details >
Needs approval from an approver in each of these files:

- ~~[OWNERS](https://github.com/kubernetes-sigs/karpenter/blob/main/OWNERS)~~ [jonathan-innis]

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":[]} -->
author_association: CONTRIBUTOR