Repository: kubernetes/autoscaler
Pull request #7977: refactor findScalableResourceProviderIDs in clusterapi
Author: login: elmiko
Labels: name: cncf-cla: yes
description: Indicates the PR's author has signed the CNCF CLA.
name: lgtm
description: "Looks good to me", indicates that a PR is ready to be merged.
name: area/cluster-autoscaler
name: size/L
description: Denotes a PR that changes 100-499 lines, ignoring generated files.
name: approved
description: Indicates a PR has been approved by an approver from all required OWNERS files.
name: kind/bug
description: Categorizes issue or PR as related to a bug.
name: area/provider/cluster-api
description: Issues or PRs related to Cluster API provider
Created: 2025-03-25T18:34:43Z | Merged: 2025-03-26T17:22:44Z



#### What type of PR is this?

/kind bug

#### What this PR does / why we need it:

this change refactors the function so that it each distinct machine state can be filtered more easily. the unit tests have been supplemented, but not changed to ensure that the functionality continues to work as expected. these changes are to help better detect edge cases where machines can be transiting through pending phase and might be removed by the autoscaler.

#### Which issue(s) this PR fixes:
<!--
*Automatically closes linked issue when PR is merged.
Usage: `Fixes #<issue number>`, or `Fixes (paste link of issue)`.
_If PR is about `failing-tests or flakes`, please post the related issues/tests in a comment and do not use `Fixes`_*
-->
Fixes #7976 

#### Special notes for your reviewer:

#### Does this PR introduce a user-facing change?
<!--
If no, just write "NONE" in the release-note block below.
If yes, a release note is required:
Enter your extended release note in the block below. If the PR requires additional action from users switching to the new release, include the string "action required".

For more information on release notes see: https://git.k8s.io/community/contributors/guide/release-notes.md
-->
```release-note
NONE
```

#### Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

<!--
This section can be blank if this pull request does not require a release note.

When adding links which point to resources within git repositories, like
KEPs or supporting documentation, please reference a specific commit and avoid
linking directly to the master branch. This ensures that links reference a
specific point in time, rather than a document that may change over time.

See here for guidance on getting permanent links to files: https://help.github.com/en/articles/getting-permanent-links-to-files

Please use the following format for linking documentation:
- [KEP]: <link>
- [Usage]: <link>
- [Other doc]: <link>
-->
```docs

```


Linked issue #7976: clusterapi provider can fail to decrease target size when machines have a provider id but fail to become a node
**Which component are you using?**:

/area cluster-autoscaler

**What version of the component are you using?**:


Component version: master

**What k8s version are you using (`kubectl version`)?**: tested with 1.29, 1.30, 1.31

**What environment is this in?**:

clusterapi

**What did you expect to happen?**:

machines that have a provider ID but have been stopped by the cloud provider should not prevent the autoscaler from decreasing the target size of a node group.

**What happened instead?**:

autoscaler fails to reduce the node group, with the following error in the logs:

```
static_autoscaler.go:448] Failed to fix node group sizes: failed to decrease MachineSet/<redacted>: node group <redacted>: attempt to delete existing nodes currentReplicas:10 delta:-1 existingNodes: 10
```

**How to reproduce it (as minimally and precisely as possible)**:

this is difficult to reproduce but will happen in cases where spot instances are used, with provider IDs given, and then stopped by the infrastructure before becoming a node.

**Anything else we need to know?**:

this is related to #7928 

Diff:
diff --git a/cluster-autoscaler/cloudprovider/clusterapi/clusterapi_controller.go b/cluster-autoscaler/cloudprovider/clusterapi/clusterapi_controller.go
index 6c95ca23fc03..bf6cca922671 100644
--- a/cluster-autoscaler/cloudprovider/clusterapi/clusterapi_controller.go
+++ b/cluster-autoscaler/cloudprovider/clusterapi/clusterapi_controller.go
@@ -638,6 +638,7 @@ func (c *machineController) findScalableResourceProviderIDs(scalableResource *un
 	}
 
 	for _, machine := range machines {
+		// Failed Machines
 		// In some cases it is possible for a machine to have acquired a provider ID from the infrastructure and
 		// then become failed later. We want to ensure that a failed machine is not counted towards the total
 		// number of nodes in the cluster, for this reason we will detect a failed machine first, regardless
@@ -648,20 +649,53 @@ func (c *machineController) findScalableResourceProviderIDs(scalableResource *un
 		}
 
 		if found {
-			klog.V(4).Infof("Status.FailureMessage of machine %q is %q", machine.GetName(), failureMessage)
-			// Provide a fake ID to allow the autoscaler to track machines that will never
+			// Provide a normalized ID to allow the autoscaler to track machines that will never
 			// become nodes and mark the nodegroup unhealthy after maxNodeProvisionTime.
 			// Fake ID needs to be recognised later and converted into a machine key.
 			// Use an underscore as a separator between namespace and name as it is not a
 			// valid character within a namespace name.
+			klog.V(4).Infof("Status.FailureMessage of machine %q is %q", machine.GetName(), failureMessage)
 			providerIDs = append(providerIDs, createFailedMachineNormalizedProviderID(machine.GetNamespace(), machine.GetName()))
 			continue
 		}
 
-		// Next we check for the provider ID. Machines with a provider ID can fall into one of a few
-		// categories: creating, running, deleting, and sometimes failed (but we filtered those earlier).
-		// Depending on which category the machine is in, it might have a deleting or pending guard
-		// added to it's provider ID so that we can later properly filter machines.
+		// Deleting Machines
+		// Machines that are in deleting state should be identified so that in scenarios where the core
+		// autoscaler would like to adjust the size of a node group, we can give a proper count and
+		// be able to filter machines in that state, regardless of whether they are still active nodes in the cluster.
+		// We give these machines normalized provider IDs to aid in the filtering process.
+		if !machine.GetDeletionTimestamp().IsZero() {
+			klog.V(4).Infof("Machine %q has a non-zero deletion timestamp", machine.GetName())
+			providerIDs = append(providerIDs, createDeletingMachineNormalizedProviderID(machine.GetNamespace(), machine.GetName()))
+			continue
+		}
+
+		// Pending Machines
+		// Machines that do not yet have an associated node reference are considering to be pending. These
+		// nodes need to be filtered so that in a case where a machine is not becoming a node, or the instance
+		// lifecycle has changed during provisioning (eg spot instance going away), or the core autoscaler has
+		// decided that the node is not needed.
+		// Look for a node reference in the status, a machine without a node reference, and that is also not
+		// in failed or deleting state, has not yet become a node, and should be marked as pending.
+		// We give these machines normalized provider IDs to aid in the filtering process.
+		_, found, err = unstructured.NestedFieldCopy(machine.UnstructuredContent(), "status", "nodeRef")
+		if err != nil {
+			return nil, err
+		}
+
+		if !found {
+			klog.V(4).Infof("Status.NodeRef of machine %q is currently nil", machine.GetName())
+			providerIDs = append(providerIDs, createPendingMachineProviderID(machine.GetNamespace(), machine.GetName()))
+			continue
+		}
+
+		// Running Machines
+		// We have filtered out the machines in failed, deleting, and pending states. We now check the provider
+		// ID and potentially the node reference details. It is ok for a machine not to have a provider ID as
+		// not all CAPI provider implement this field, but a machine in running state should have a valid
+		// node reference. If a provider ID is present, we add that to the list as we know it is not failed,
+		// deleting, or pending. If an empty provider ID is present, we check the node details to ensure that
+		// the machine references a valid node.
 		providerID, found, err := unstructured.NestedString(machine.UnstructuredContent(), "spec", "providerID")
 		if err != nil {
 			return nil, err
@@ -669,12 +703,7 @@ func (c *machineController) findScalableResourceProviderIDs(scalableResource *un
 
 		if found {
 			if providerID != "" {
-				// If the machine is deleting, prepend the deletion guard on the provider id
-				// so that it can be properly filtered when counting the number of nodes and instances.
-				if !machine.GetDeletionTimestamp().IsZero() {
-					klog.V(4).Infof("Machine %q has a non-zero deletion timestamp", machine.GetName())
-					providerID = createDeletingMachineNormalizedProviderID(machine.GetNamespace(), machine.GetName())
-				}
+				// Machine has a provider ID, add it to the list
 				providerIDs = append(providerIDs, providerID)
 				continue
 			}
@@ -682,19 +711,7 @@ func (c *machineController) findScalableResourceProviderIDs(scalableResource *un
 
 		klog.Warningf("Machine %q has no providerID", machine.GetName())
 
-		// Look for a node reference in the status, a machine with a provider ID but no node reference has not
-		// yet become a node and should be marked as pending.
-		_, found, err = unstructured.NestedFieldCopy(machine.UnstructuredContent(), "status", "nodeRef")
-		if err != nil {
-			return nil, err
-		}
-
-		if !found {
-			klog.V(4).Infof("Status.NodeRef of machine %q is currently nil", machine.GetName())
-			providerIDs = append(providerIDs, createPendingMachineProviderID(machine.GetNamespace(), machine.GetName()))
-			continue
-		}
-
+		// Begin checking to determine if the node reference is valid
 		nodeRefKind, found, err := unstructured.NestedString(machine.UnstructuredContent(), "status", "nodeRef", "kind")
 		if err != nil {
 			return nil, err
@@ -716,6 +733,8 @@ func (c *machineController) findScalableResourceProviderIDs(scalableResource *un
 				return nil, fmt.Errorf("unknown node %q", nodeRefName)
 			}
 
+			// A node has been found that corresponds to this machine, since we know that this machine has
+			// an empty provider ID, we add the provider ID from the node to the list.
 			if node != nil {
 				providerIDs = append(providerIDs, node.Spec.ProviderID)
 			}
diff --git a/cluster-autoscaler/cloudprovider/clusterapi/clusterapi_controller_test.go b/cluster-autoscaler/cloudprovider/clusterapi/clusterapi_controller_test.go
index 501f89da17dc..fadc07aa7836 100644
--- a/cluster-autoscaler/cloudprovider/clusterapi/clusterapi_controller_test.go
+++ b/cluster-autoscaler/cloudprovider/clusterapi/clusterapi_controller_test.go
@@ -1335,19 +1335,6 @@ func TestControllerMachineSetNodeNamesUsingProviderID(t *testing.T) {
 	controller, stop := mustCreateTestController(t, testConfig)
 	defer stop()
 
-	// Remove Status.NodeRef.Name on all the machines. We want to
-	// force machineSetNodeNames() to only consider the provider
-	// ID for lookups.
-	for i := range testConfig.machines {
-		machine := testConfig.machines[i].DeepCopy()
-
-		unstructured.RemoveNestedField(machine.Object, "status", "nodeRef")
-
-		if err := updateResource(controller.managementClient, controller.machineInformer, controller.machineResource, machine); err != nil {
-			t.Fatalf("unexpected error updating machine, got %v", err)
-		}
-	}
-
 	nodegroups, err := controller.nodeGroups()
 	if err != nil {
 		t.Fatalf("unexpected error: %v", err)
diff --git a/cluster-autoscaler/cloudprovider/clusterapi/clusterapi_nodegroup_test.go b/cluster-autoscaler/cloudprovider/clusterapi/clusterapi_nodegroup_test.go
index 11bcec4e3df7..75ec29805ba6 100644
--- a/cluster-autoscaler/cloudprovider/clusterapi/clusterapi_nodegroup_test.go
+++ b/cluster-autoscaler/cloudprovider/clusterapi/clusterapi_nodegroup_test.go
@@ -406,16 +406,18 @@ func TestNodeGroupIncreaseSize(t *testing.T) {
 
 func TestNodeGroupDecreaseTargetSize(t *testing.T) {
 	type testCase struct {
-		description                        string
-		delta                              int
-		initial                            int32
-		targetSizeIncrement                int32
-		expected                           int32
-		expectedError                      bool
-		includeDeletingMachine             bool
-		includeFailedMachine               bool
-		includeFailedMachineWithProviderID bool
-		includePendingMachine              bool
+		description                         string
+		delta                               int
+		initial                             int32
+		targetSizeIncrement                 int32
+		expected                            int32
+		expectedError                       bool
+		includeDeletingMachine              bool
+		includeFailedMachine                bool
+		includeFailedMachineWithProviderID  bool
+		includePendingMachine               bool
+		includePendingMachineWithProviderID bool
+		machinesDoNotHaveProviderIDs        bool
 	}
 
 	test := func(t *testing.T, tc *testCase, testConfig *testConfig) {
@@ -468,7 +470,9 @@ func TestNodeGroupDecreaseTargetSize(t *testing.T) {
 			// Simulate a pending machine
 			machine := testConfig.machines[2].DeepCopy()
 
-			unstructured.RemoveNestedField(machine.Object, "spec", "providerID")
+			if !tc.includePendingMachineWithProviderID {
+				unstructured.RemoveNestedField(machine.Object, "spec", "providerID")
+			}
 			unstructured.RemoveNestedField(machine.Object, "status", "nodeRef")
 
 			if err := updateResource(controller.managementClient, controller.machineInformer, controller.machineResource, machine); err != nil {
@@ -476,6 +480,17 @@ func TestNodeGroupDecreaseTargetSize(t *testing.T) {
 			}
 		}
 
+		// machines with no provider id can be created on some providers, notably bare metal
+		if tc.machinesDoNotHaveProviderIDs {
+			for _, machine := range testConfig.machines {
+				updated := machine.DeepCopy()
+				unstructured.RemoveNestedField(updated.Object, "spec", "providerID")
+				if err := updateResource(controller.managementClient, controller.machineInformer, controller.machineResource, updated); err != nil {
+					t.Fatalf("unexpected error updating machine, got %v", err)
+				}
+			}
+		}
+
 		nodegroups, err := controller.nodeGroups()
 		if err != nil {
 			t.Fatalf("unexpected error: %v", err)
@@ -648,6 +663,23 @@ func TestNodeGroupDecreaseTargetSize(t *testing.T) {
 			includeFailedMachine:               true,
 			includeFailedMachineWithProviderID: true,
 		},
+		{
+			description:                         "A node group with 4 replicas with one pending machine that has a provider ID should decrease by 1",
+			initial:                             4,
+			targetSizeIncrement:                 0,
+			expected:                            3,
+			delta:                               -1,
+			includePendingMachine:               true,
+			includePendingMachineWithProviderID: true,
+		},
+		{
+			description:                  "A node group with target size 4 but only 3 existing instances without provider IDs should not scale out",
+			initial:                      3,
+			targetSizeIncrement:          1,
+			expected:                     3,
+			delta:                        -1,
+			machinesDoNotHaveProviderIDs: true,
+		},
 	}
 
 	annotations := map[string]string{


Reviews:
user: login: jackfrancis
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-26T15:23:17Z
user: login: jackfrancis
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-26T15:26:11Z
user: login: jackfrancis
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-26T15:31:01Z
user: login: elmiko
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-26T15:48:25Z
user: login: elmiko
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-26T15:49:57Z
user: login: elmiko
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-26T15:51:02Z
user: login: jackfrancis
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-26T16:16:07Z
user: login: jackfrancis
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-26T16:27:10Z
user: login: jackfrancis
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-26T16:28:46Z
user: login: elmiko
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-26T16:39:36Z
user: login: elmiko
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-26T16:40:31Z
user: login: elmiko
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-26T16:41:18Z

Review comments:
diff_hunk: @@ -1335,19 +1335,6 @@ func TestControllerMachineSetNodeNamesUsingProviderID(t *testing.T) {
 	controller, stop := mustCreateTestController(t, testConfig)
 	defer stop()
 
-	// Remove Status.NodeRef.Name on all the machines. We want to
path: cluster-autoscaler/cloudprovider/clusterapi/clusterapi_controller_test.go
user: login: jackfrancis
body: Is the idea here that the business logic has now been updated to properly prioritize providerID over Name, and so we don't need to babysit the data input like this anymore?
created_at: 2025-03-26T15:23:17Z
author_association: CONTRIBUTOR
diff_hunk: @@ -648,53 +649,69 @@ func (c *machineController) findScalableResourceProviderIDs(scalableResource *un
 		}
 
 		if found {
-			klog.V(4).Infof("Status.FailureMessage of machine %q is %q", machine.GetName(), failureMessage)
-			// Provide a fake ID to allow the autoscaler to track machines that will never
+			// Provide a normalized ID to allow the autoscaler to track machines that will never
 			// become nodes and mark the nodegroup unhealthy after maxNodeProvisionTime.
 			// Fake ID needs to be recognised later and converted into a machine key.
 			// Use an underscore as a separator between namespace and name as it is not a
 			// valid character within a namespace name.
+			klog.V(4).Infof("Status.FailureMessage of machine %q is %q", machine.GetName(), failureMessage)
 			providerIDs = append(providerIDs, createFailedMachineNormalizedProviderID(machine.GetNamespace(), machine.GetName()))
 			continue
 		}
 
-		// Next we check for the provider ID. Machines with a provider ID can fall into one of a few
-		// categories: creating, running, deleting, and sometimes failed (but we filtered those earlier).
-		// Depending on which category the machine is in, it might have a deleting or pending guard
-		// added to it's provider ID so that we can later properly filter machines.
+		// Deleting Machines
+		// Machines that are in deleting state should be identified so that in scenarios where the core
+		// autoscaler would like to adjust the size of a node group, we can give a proper count and
+		// be able to filter machines in that state, regardless of whether they become a node or not.
path: cluster-autoscaler/cloudprovider/clusterapi/clusterapi_controller.go
user: login: jackfrancis
body: I'm not sure a "Deleting machine" would ever become a node, would

> regardless of whether or they are still active nodes in the cluster

be better than

> regardless of whether they become a node or not.
created_at: 2025-03-26T15:26:11Z
author_association: CONTRIBUTOR
diff_hunk: @@ -648,53 +649,69 @@ func (c *machineController) findScalableResourceProviderIDs(scalableResource *un
 		}
 
 		if found {
-			klog.V(4).Infof("Status.FailureMessage of machine %q is %q", machine.GetName(), failureMessage)
-			// Provide a fake ID to allow the autoscaler to track machines that will never
+			// Provide a normalized ID to allow the autoscaler to track machines that will never
 			// become nodes and mark the nodegroup unhealthy after maxNodeProvisionTime.
 			// Fake ID needs to be recognised later and converted into a machine key.
 			// Use an underscore as a separator between namespace and name as it is not a
 			// valid character within a namespace name.
+			klog.V(4).Infof("Status.FailureMessage of machine %q is %q", machine.GetName(), failureMessage)
 			providerIDs = append(providerIDs, createFailedMachineNormalizedProviderID(machine.GetNamespace(), machine.GetName()))
 			continue
 		}
 
-		// Next we check for the provider ID. Machines with a provider ID can fall into one of a few
-		// categories: creating, running, deleting, and sometimes failed (but we filtered those earlier).
-		// Depending on which category the machine is in, it might have a deleting or pending guard
-		// added to it's provider ID so that we can later properly filter machines.
+		// Deleting Machines
+		// Machines that are in deleting state should be identified so that in scenarios where the core
+		// autoscaler would like to adjust the size of a node group, we can give a proper count and
+		// be able to filter machines in that state, regardless of whether they become a node or not.
+		// We give these machines normalized provider IDs to aid in the filtering process.
+		if !machine.GetDeletionTimestamp().IsZero() {
+			klog.V(4).Infof("Machine %q has a non-zero deletion timestamp", machine.GetName())
+			providerIDs = append(providerIDs, createDeletingMachineNormalizedProviderID(machine.GetNamespace(), machine.GetName()))
+			continue
+		}
+
+		// Pending Machines
+		// Machines that do not yet have an associated node reference are considering to be pending. These
+		// nodes need to be filtered so that in a case where a machine is not becoming a node, or the instance
+		// lifecycle has changed during provisioning (eg spot instance going away), or the core autoscaler has
+		// decided that the node is not needed.
+		// Look for a node reference in the status, a machine without a node reference, and that is also not
+		// in failed or deleting state, has not yet become a node, and should be marked as pending.
+		// We give these machines normalized provider IDs to aid in the filtering process.
+		_, found, err = unstructured.NestedFieldCopy(machine.UnstructuredContent(), "status", "nodeRef")
+		if err != nil {
+			return nil, err
+		}
+
+		if !found {
+			klog.V(4).Infof("Status.NodeRef of machine %q is currently nil", machine.GetName())
+			providerIDs = append(providerIDs, createPendingMachineProviderID(machine.GetNamespace(), machine.GetName()))
+			continue
+		}
+
+		// Running Machines
+		// We have filtered out the machines in failed, deleting, and pending states. We now check the provider
+		// ID and potentially the node reference details. It is ok for a machine not to have a provider ID as
+		// not all CAPI provider implement this field, but a machine in running state should have a valid
+		// node reference. If a provider ID is present, we add that to the list as we know it is not failed,
+		// deleting, or pending. If an empty provider ID is present, we check the node details to ensure that
+		// the machine references a valid node.
 		providerID, found, err := unstructured.NestedString(machine.UnstructuredContent(), "spec", "providerID")
 		if err != nil {
 			return nil, err
 		}
 
 		if found {
 			if providerID != "" {
-				// If the machine is deleting, prepend the deletion guard on the provider id
-				// so that it can be properly filtered when counting the number of nodes and instances.
-				if !machine.GetDeletionTimestamp().IsZero() {
-					klog.V(4).Infof("Machine %q has a non-zero deletion timestamp", machine.GetName())
-					providerID = createDeletingMachineNormalizedProviderID(machine.GetNamespace(), machine.GetName())
-				}
+				// Machine has a provider ID, add it to the list
 				providerIDs = append(providerIDs, providerID)
 				continue
 			}
 		}
 
 		klog.Warningf("Machine %q has no providerID", machine.GetName())
 
-		// Look for a node reference in the status, a machine with a provider ID but no node reference has not
-		// yet become a node and should be marked as pending.
-		_, found, err = unstructured.NestedFieldCopy(machine.UnstructuredContent(), "status", "nodeRef")
-		if err != nil {
-			return nil, err
-		}
-
-		if !found {
-			klog.V(4).Infof("Status.NodeRef of machine %q is currently nil", machine.GetName())
-			providerIDs = append(providerIDs, createPendingMachineProviderID(machine.GetNamespace(), machine.GetName()))
-			continue
-		}
-
+		// Begin checking to determine if the node reference is valid
path: cluster-autoscaler/cloudprovider/clusterapi/clusterapi_controller.go
user: login: jackfrancis
body: does the `found` response from `unstructured.NestedString(machine.UnstructuredContent(), "status", "nodeRef", "kind")` (L715) need to be taken into consideration before we call `unstructured.NestedString(machine.UnstructuredContent(), "status", "nodeRef", "name")` (L725)?

I see we're considering it in L720 (`if found && nodeRefKind != "Node" {`), but is that the only place we want to evaluate it?
created_at: 2025-03-26T15:31:01Z
author_association: CONTRIBUTOR
diff_hunk: @@ -648,53 +649,69 @@ func (c *machineController) findScalableResourceProviderIDs(scalableResource *un
 		}
 
 		if found {
-			klog.V(4).Infof("Status.FailureMessage of machine %q is %q", machine.GetName(), failureMessage)
-			// Provide a fake ID to allow the autoscaler to track machines that will never
+			// Provide a normalized ID to allow the autoscaler to track machines that will never
 			// become nodes and mark the nodegroup unhealthy after maxNodeProvisionTime.
 			// Fake ID needs to be recognised later and converted into a machine key.
 			// Use an underscore as a separator between namespace and name as it is not a
 			// valid character within a namespace name.
+			klog.V(4).Infof("Status.FailureMessage of machine %q is %q", machine.GetName(), failureMessage)
 			providerIDs = append(providerIDs, createFailedMachineNormalizedProviderID(machine.GetNamespace(), machine.GetName()))
 			continue
 		}
 
-		// Next we check for the provider ID. Machines with a provider ID can fall into one of a few
-		// categories: creating, running, deleting, and sometimes failed (but we filtered those earlier).
-		// Depending on which category the machine is in, it might have a deleting or pending guard
-		// added to it's provider ID so that we can later properly filter machines.
+		// Deleting Machines
+		// Machines that are in deleting state should be identified so that in scenarios where the core
+		// autoscaler would like to adjust the size of a node group, we can give a proper count and
+		// be able to filter machines in that state, regardless of whether they become a node or not.
path: cluster-autoscaler/cloudprovider/clusterapi/clusterapi_controller.go
user: login: elmiko
body: yes, will update.
created_at: 2025-03-26T15:48:25Z
author_association: CONTRIBUTOR
diff_hunk: @@ -648,53 +649,69 @@ func (c *machineController) findScalableResourceProviderIDs(scalableResource *un
 		}
 
 		if found {
-			klog.V(4).Infof("Status.FailureMessage of machine %q is %q", machine.GetName(), failureMessage)
-			// Provide a fake ID to allow the autoscaler to track machines that will never
+			// Provide a normalized ID to allow the autoscaler to track machines that will never
 			// become nodes and mark the nodegroup unhealthy after maxNodeProvisionTime.
 			// Fake ID needs to be recognised later and converted into a machine key.
 			// Use an underscore as a separator between namespace and name as it is not a
 			// valid character within a namespace name.
+			klog.V(4).Infof("Status.FailureMessage of machine %q is %q", machine.GetName(), failureMessage)
 			providerIDs = append(providerIDs, createFailedMachineNormalizedProviderID(machine.GetNamespace(), machine.GetName()))
 			continue
 		}
 
-		// Next we check for the provider ID. Machines with a provider ID can fall into one of a few
-		// categories: creating, running, deleting, and sometimes failed (but we filtered those earlier).
-		// Depending on which category the machine is in, it might have a deleting or pending guard
-		// added to it's provider ID so that we can later properly filter machines.
+		// Deleting Machines
+		// Machines that are in deleting state should be identified so that in scenarios where the core
+		// autoscaler would like to adjust the size of a node group, we can give a proper count and
+		// be able to filter machines in that state, regardless of whether they become a node or not.
+		// We give these machines normalized provider IDs to aid in the filtering process.
+		if !machine.GetDeletionTimestamp().IsZero() {
+			klog.V(4).Infof("Machine %q has a non-zero deletion timestamp", machine.GetName())
+			providerIDs = append(providerIDs, createDeletingMachineNormalizedProviderID(machine.GetNamespace(), machine.GetName()))
+			continue
+		}
+
+		// Pending Machines
+		// Machines that do not yet have an associated node reference are considering to be pending. These
+		// nodes need to be filtered so that in a case where a machine is not becoming a node, or the instance
+		// lifecycle has changed during provisioning (eg spot instance going away), or the core autoscaler has
+		// decided that the node is not needed.
+		// Look for a node reference in the status, a machine without a node reference, and that is also not
+		// in failed or deleting state, has not yet become a node, and should be marked as pending.
+		// We give these machines normalized provider IDs to aid in the filtering process.
+		_, found, err = unstructured.NestedFieldCopy(machine.UnstructuredContent(), "status", "nodeRef")
+		if err != nil {
+			return nil, err
+		}
+
+		if !found {
+			klog.V(4).Infof("Status.NodeRef of machine %q is currently nil", machine.GetName())
+			providerIDs = append(providerIDs, createPendingMachineProviderID(machine.GetNamespace(), machine.GetName()))
+			continue
+		}
+
+		// Running Machines
+		// We have filtered out the machines in failed, deleting, and pending states. We now check the provider
+		// ID and potentially the node reference details. It is ok for a machine not to have a provider ID as
+		// not all CAPI provider implement this field, but a machine in running state should have a valid
+		// node reference. If a provider ID is present, we add that to the list as we know it is not failed,
+		// deleting, or pending. If an empty provider ID is present, we check the node details to ensure that
+		// the machine references a valid node.
 		providerID, found, err := unstructured.NestedString(machine.UnstructuredContent(), "spec", "providerID")
 		if err != nil {
 			return nil, err
 		}
 
 		if found {
 			if providerID != "" {
-				// If the machine is deleting, prepend the deletion guard on the provider id
-				// so that it can be properly filtered when counting the number of nodes and instances.
-				if !machine.GetDeletionTimestamp().IsZero() {
-					klog.V(4).Infof("Machine %q has a non-zero deletion timestamp", machine.GetName())
-					providerID = createDeletingMachineNormalizedProviderID(machine.GetNamespace(), machine.GetName())
-				}
+				// Machine has a provider ID, add it to the list
 				providerIDs = append(providerIDs, providerID)
 				continue
 			}
 		}
 
 		klog.Warningf("Machine %q has no providerID", machine.GetName())
 
-		// Look for a node reference in the status, a machine with a provider ID but no node reference has not
-		// yet become a node and should be marked as pending.
-		_, found, err = unstructured.NestedFieldCopy(machine.UnstructuredContent(), "status", "nodeRef")
-		if err != nil {
-			return nil, err
-		}
-
-		if !found {
-			klog.V(4).Infof("Status.NodeRef of machine %q is currently nil", machine.GetName())
-			providerIDs = append(providerIDs, createPendingMachineProviderID(machine.GetNamespace(), machine.GetName()))
-			continue
-		}
-
+		// Begin checking to determine if the node reference is valid
path: cluster-autoscaler/cloudprovider/clusterapi/clusterapi_controller.go
user: login: elmiko
body: yes, i think we just need to check it that one time. i left this part of the logic unchanged as it seemed to be correct.
created_at: 2025-03-26T15:49:57Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1335,19 +1335,6 @@ func TestControllerMachineSetNodeNamesUsingProviderID(t *testing.T) {
 	controller, stop := mustCreateTestController(t, testConfig)
 	defer stop()
 
-	// Remove Status.NodeRef.Name on all the machines. We want to
path: cluster-autoscaler/cloudprovider/clusterapi/clusterapi_controller_test.go
user: login: elmiko
body: correct, in the previous iteration, there was a clause to check the provider id and then do different things based on having a provider id and other stuff. since i refactored to be more distinct about failed, deleting, and pending machines, this deep testing of the conditional is not needed.
created_at: 2025-03-26T15:51:02Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1335,19 +1335,6 @@ func TestControllerMachineSetNodeNamesUsingProviderID(t *testing.T) {
 	controller, stop := mustCreateTestController(t, testConfig)
 	defer stop()
 
-	// Remove Status.NodeRef.Name on all the machines. We want to
path: cluster-autoscaler/cloudprovider/clusterapi/clusterapi_controller_test.go
user: login: jackfrancis
body: ack, this is a significant improvement in edge case detection, stuff like this in UT is essentially "let's get rid of any possible edge cases because that might be harder to verify" ;)
created_at: 2025-03-26T16:16:06Z
author_association: CONTRIBUTOR
diff_hunk: @@ -648,6 +663,23 @@ func TestNodeGroupDecreaseTargetSize(t *testing.T) {
 			includeFailedMachine:               true,
 			includeFailedMachineWithProviderID: true,
 		},
+		{
+			description:                         "A node group with 4 replicas with one pending machine that has a provider ID should decrease by 1",
+			initial:                             4,
+			targetSizeIncrement:                 0,
+			expected:                            3,
+			delta:                               -1,
+			includePendingMachine:               true,
+			includePendingMachineWithProviderID: true,
+		},
+		{
+			description:                  "A node group with target size 4 but only 3 existing instances without provider IDs should decrease by 1",
path: cluster-autoscaler/cloudprovider/clusterapi/clusterapi_nodegroup_test.go
user: login: jackfrancis
body: let's s/should decrease by 1/should not scale out/
created_at: 2025-03-26T16:27:10Z
author_association: CONTRIBUTOR
diff_hunk: @@ -406,16 +406,18 @@ func TestNodeGroupIncreaseSize(t *testing.T) {
 
 func TestNodeGroupDecreaseTargetSize(t *testing.T) {
 	type testCase struct {
-		description                        string
-		delta                              int
-		initial                            int32
-		targetSizeIncrement                int32
-		expected                           int32
-		expectedError                      bool
-		includeDeletingMachine             bool
-		includeFailedMachine               bool
-		includeFailedMachineWithProviderID bool
-		includePendingMachine              bool
+		description                         string
+		delta                               int
+		initial                             int32
+		targetSizeIncrement                 int32
+		expected                            int32
+		expectedError                       bool
+		includeDeletingMachine              bool
+		includeFailedMachine                bool
+		includeFailedMachineWithProviderID  bool
+		includePendingMachine               bool
+		includePendingMachineWithProviderID bool
path: cluster-autoscaler/cloudprovider/clusterapi/clusterapi_nodegroup_test.go
user: login: jackfrancis
body: just a note that these UT scenarios could probably be improved in the future by allowing a more flexible count of machines with particular states (right now we only organize our scenarios so they have one of each type of [not sure about the combinatorics] failed/pending/no providerID)
created_at: 2025-03-26T16:28:46Z
author_association: CONTRIBUTOR
diff_hunk: @@ -406,16 +406,18 @@ func TestNodeGroupIncreaseSize(t *testing.T) {
 
 func TestNodeGroupDecreaseTargetSize(t *testing.T) {
 	type testCase struct {
-		description                        string
-		delta                              int
-		initial                            int32
-		targetSizeIncrement                int32
-		expected                           int32
-		expectedError                      bool
-		includeDeletingMachine             bool
-		includeFailedMachine               bool
-		includeFailedMachineWithProviderID bool
-		includePendingMachine              bool
+		description                         string
+		delta                               int
+		initial                             int32
+		targetSizeIncrement                 int32
+		expected                            int32
+		expectedError                       bool
+		includeDeletingMachine              bool
+		includeFailedMachine                bool
+		includeFailedMachineWithProviderID  bool
+		includePendingMachine               bool
+		includePendingMachineWithProviderID bool
path: cluster-autoscaler/cloudprovider/clusterapi/clusterapi_nodegroup_test.go
user: login: elmiko
body: yes, what i want to do as a next step is to change our unit test infrastructure to have a more fluent style builder for the capi objects. i think that will make the tests easier to understand. this is a middle ground.
created_at: 2025-03-26T16:39:36Z
author_association: CONTRIBUTOR
diff_hunk: @@ -406,16 +406,18 @@ func TestNodeGroupIncreaseSize(t *testing.T) {
 
 func TestNodeGroupDecreaseTargetSize(t *testing.T) {
 	type testCase struct {
-		description                        string
-		delta                              int
-		initial                            int32
-		targetSizeIncrement                int32
-		expected                           int32
-		expectedError                      bool
-		includeDeletingMachine             bool
-		includeFailedMachine               bool
-		includeFailedMachineWithProviderID bool
-		includePendingMachine              bool
+		description                         string
+		delta                               int
+		initial                             int32
+		targetSizeIncrement                 int32
+		expected                            int32
+		expectedError                       bool
+		includeDeletingMachine              bool
+		includeFailedMachine                bool
+		includeFailedMachineWithProviderID  bool
+		includePendingMachine               bool
+		includePendingMachineWithProviderID bool
path: cluster-autoscaler/cloudprovider/clusterapi/clusterapi_nodegroup_test.go
user: login: elmiko
body: as to the combinatorics, i have tried to make it so that we can test combinations of machines. i would like to add that during the next step.
created_at: 2025-03-26T16:40:31Z
author_association: CONTRIBUTOR
diff_hunk: @@ -648,6 +663,23 @@ func TestNodeGroupDecreaseTargetSize(t *testing.T) {
 			includeFailedMachine:               true,
 			includeFailedMachineWithProviderID: true,
 		},
+		{
+			description:                         "A node group with 4 replicas with one pending machine that has a provider ID should decrease by 1",
+			initial:                             4,
+			targetSizeIncrement:                 0,
+			expected:                            3,
+			delta:                               -1,
+			includePendingMachine:               true,
+			includePendingMachineWithProviderID: true,
+		},
+		{
+			description:                  "A node group with target size 4 but only 3 existing instances without provider IDs should decrease by 1",
path: cluster-autoscaler/cloudprovider/clusterapi/clusterapi_nodegroup_test.go
user: login: elmiko
body: updated
created_at: 2025-03-26T16:41:18Z
author_association: CONTRIBUTOR

PR comments:
user: login: k8s-ci-robot
created_at: 2025-03-25T18:34:51Z
body: [APPROVALNOTIFIER] This PR is **APPROVED**

This pull-request has been approved by: *<a href="https://github.com/kubernetes/autoscaler/pull/7977#" title="Author self-approved">elmiko</a>*

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=kubernetes%2Fautoscaler).

The pull request process is described [here](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process)

<details >
Needs approval from an approver in each of these files:

- ~~[cluster-autoscaler/cloudprovider/clusterapi/OWNERS](https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/clusterapi/OWNERS)~~ [elmiko]

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":[]} -->
author_association: CONTRIBUTOR
user: login: elmiko
created_at: 2025-03-25T20:44:36Z
body: the failing test is trying to do something that we don't necessarily want. it's removing the node ref from the machine object to force a lookup of the provider id from the machine instead of the node. but, we are changing how the nodes are listed from the provider to ensure that a node without a noderef, but with a provider id, is properly identified as pending.

 i am going to improve these tests to ensure we are getting all the possible outcomes.
author_association: CONTRIBUTOR
user: login: elmiko
created_at: 2025-03-25T21:44:37Z
body: after reviewing the [failing test](https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/clusterapi/clusterapi_controller_test.go#L1381-L1430), i think we can safely remove this as it is testing functionality that does not comport with the current understanding of how the `Nodes` function should be working.

the test relies on the idea that a Machine object with no `.status.nodeRef.name` will be able to be found by examining the node object and then finding the provider id. it references a function called `machineSetNodeNames` in the comments; this function does not exist. we have changed the functionality of the `Nodes` function to consider Machines without a node ref to be in pending state. due to this change, the provider id for the Machine is returned as normalized with the pending label prefixed. i have changed the test to not remove the noderef as the logic would not allow a machine with no noderef to pass beyond the pending guard.
author_association: CONTRIBUTOR
user: login: elmiko
created_at: 2025-03-26T16:38:42Z
body: updated with doc string suggestion
author_association: CONTRIBUTOR
user: login: jackfrancis
created_at: 2025-03-26T17:22:14Z
body: /lgtm
author_association: CONTRIBUTOR