Repository: gardener/machine-controller-manager
Pull request #972: Trigger removal of CA tainted machine without waiting for replica change
Author: login: elankath
Labels: name: needs/review
description: Needs review
name: needs/ok-to-test
description: Needs approval for testing (check PR in detail before setting this label because PR is run on CI/CD)
name: status/closed
description: Issue is closed (either delivered or triaged)
name: size/M
description: Denotes a PR that changes 30-99 lines, ignoring generated files.
Created: 2025-03-10T07:40:52Z | Merged: 2025-03-21T05:46:45Z

**What this PR does / why we need it**:

**Which issue(s) this PR fixes**:
Fixes #971

**Special notes for your reviewer**:

**Release note**:
<!--  Write your release note:
1. Enter your release note in the below block.
2. If no release note is required, just write "NONE" within the block.

Format of block header: <category> <target_group>
Possible values:
- category:       breaking|feature|bugfix|doc|other
- target_group:   user|operator|developer|dependency
-->
```bugfix operator
CA tainted node is removed as soon as possible by MachineSet controller 
```


Linked issue #971: kube-proxy pod not healthy - scale down CA tainted node ASAP
**How to categorize this issue?**
<!--
Please select area, kind, and priority for this issue. This helps the community categorizing it.
Replace below TODOs or exchange the existing identifiers with those that fit best in your opinion.
If multiple identifiers make sense you can also state the commands multiple times, e.g.
  /area control-plane
  /area auto-scaling
  ...

"/area" identifiers:     audit-logging|auto-scaling|backup|certification|control-plane-migration|control-plane|cost|delivery|dev-productivity|disaster-recovery|documentation|high-availability|logging|metering|monitoring|networking|open-source|ops-productivity|os|performance|quality|robustness|scalability|security|storage|testing|usability|user-management
"/kind" identifiers:     api-change|bug|cleanup|discussion|enhancement|epic|impediment|poc|post-mortem|question|regression|task|technical-debt|test
"/priority" identifiers: 1|2|3|4|5 (ordered from greatest to least)
-->
/area auto-scaling
/kind bug
/priority 2

**What happened**:
kube-proxy reports Node as not healthy when CA taints a Node for deletion. MCM does not remove this node if replica count of MachineDeployment is unchanged.

k8s gardener live issue 6101            

**What you expected to happen**:
The machine set controller should mark this node as a stale node and scale it down and scale-up a fresh machine and node to match the replica count.

**How to reproduce it (as minimally and precisely as possible)**:

Taint a k8s node with the CA deletion taint and add the machine name to the triggered for deletion annotation on the machine deployment.

**Anything else we need to know?**:

**Environment**:

- Kubernetes version (use `kubectl version`):
- Cloud provider or hardware configuration:
- Others:

Linked issue comments: user: login: gardener-robot
created_at: 2025-02-27T10:35:20Z
body: @elankath Label area/|auto-scaling does not exist.
author_association: NONE

Diff:
diff --git a/pkg/controller/controller_utils.go b/pkg/controller/controller_utils.go
index 1063199bd5..7790fee284 100644
--- a/pkg/controller/controller_utils.go
+++ b/pkg/controller/controller_utils.go
@@ -786,22 +786,6 @@ func (s ActiveMachines) Less(i, j int) bool {
 	return false
 }
 
-// IsMachineActive checks if machine was active
-func IsMachineActive(p *v1alpha1.Machine) bool {
-	if p.Status.CurrentStatus.Phase == v1alpha1.MachineFailed {
-		return false
-	} else if p.Status.CurrentStatus.Phase == v1alpha1.MachineTerminating {
-		return false
-	}
-
-	return true
-}
-
-// IsMachineFailed checks if machine has failed
-func IsMachineFailed(p *v1alpha1.Machine) bool {
-	return p.Status.CurrentStatus.Phase == v1alpha1.MachineFailed
-}
-
 // MachineKey is the function used to get the machine name from machine object
 // ToCheck : as machine-namespace does not matter
 func MachineKey(machine *v1alpha1.Machine) string {
diff --git a/pkg/controller/deployment_test.go b/pkg/controller/deployment_test.go
index ecf504a9b9..d378c8fdca 100644
--- a/pkg/controller/deployment_test.go
+++ b/pkg/controller/deployment_test.go
@@ -7,6 +7,9 @@ package controller
 import (
 	"context"
 	"errors"
+	"fmt"
+	"github.com/gardener/machine-controller-manager/pkg/util/annotations"
+	"github.com/gardener/machine-controller-manager/pkg/util/provider/machineutils"
 	"time"
 
 	machinev1 "github.com/gardener/machine-controller-manager/pkg/apis/machine/v1alpha1"
@@ -1400,8 +1403,9 @@ var _ = Describe("machineDeployment", func() {
 					Labels: map[string]string{
 						"test-label": "test-label",
 					},
-					UID:        "1234567",
-					Generation: 5,
+					Annotations: make(map[string]string),
+					UID:         "1234567",
+					Generation:  5,
 				},
 				TypeMeta: metav1.TypeMeta{
 					Kind:       "MachineDeployment",
@@ -1506,7 +1510,8 @@ var _ = Describe("machineDeployment", func() {
 						"test-label":           "test-label",
 						machinev1.NodeLabelKey: "Node1-test",
 					},
-					UID: "1234567",
+					Annotations: map[string]string{},
+					UID:         "1234567",
 					OwnerReferences: []metav1.OwnerReference{
 						{
 							Kind:       "MachineSet",
@@ -1552,7 +1557,7 @@ var _ = Describe("machineDeployment", func() {
 
 		DescribeTable("This should",
 			func(preset func(testMachineDeployment *machinev1.MachineDeployment, testMachineSet *machinev1.MachineSet),
-				postCheck func(testMachineDeployment *machinev1.MachineDeployment, testMachineSet []machinev1.MachineSet, testNode *corev1.Node) error) {
+				postCheck func(testMachineDeployment *machinev1.MachineDeployment, testMachineSet []machinev1.MachineSet, testMachines []machinev1.Machine, testNode *corev1.Node) error) {
 
 				stop := make(chan struct{})
 				preset(testMachineDeployment, testMachineSet)
@@ -1575,13 +1580,15 @@ var _ = Describe("machineDeployment", func() {
 				waitForCacheSync(stop, c)
 				actualMachineDeployment, _ := c.controlMachineClient.MachineDeployments(testNamespace).Get(context.Background(), testMachineDeployment.Name, metav1.GetOptions{})
 				actualMachineSets, _ := c.controlMachineClient.MachineSets(testNamespace).List(context.Background(), metav1.ListOptions{})
+				waitForCacheSync(stop, c)
+				actualMachines, _ := c.controlMachineClient.Machines(testNamespace).List(context.Background(), metav1.ListOptions{})
 				testNode, _ := c.targetCoreClient.CoreV1().Nodes().Get(context.Background(), testNode.Name, metav1.GetOptions{})
 
-				Expect(postCheck(actualMachineDeployment, actualMachineSets.Items, testNode)).To(BeNil())
+				Expect(postCheck(actualMachineDeployment, actualMachineSets.Items, actualMachines.Items, testNode)).To(BeNil())
 			},
 			Entry("reconcile the machineDeployment and return nil",
 				func(_ *machinev1.MachineDeployment, _ *machinev1.MachineSet) {},
-				func(_ *machinev1.MachineDeployment, _ []machinev1.MachineSet, _ *corev1.Node) error {
+				func(_ *machinev1.MachineDeployment, _ []machinev1.MachineSet, _ []machinev1.Machine, _ *corev1.Node) error {
 					return nil
 				},
 			),
@@ -1592,7 +1599,7 @@ var _ = Describe("machineDeployment", func() {
 					testMachineSet.Spec = machinev1.MachineSetSpec{}
 					testMachineSet.Status = machinev1.MachineSetStatus{}
 				},
-				func(_ *machinev1.MachineDeployment, testMachineSets []machinev1.MachineSet, _ *corev1.Node) error {
+				func(_ *machinev1.MachineDeployment, testMachineSets []machinev1.MachineSet, _ []machinev1.Machine, _ *corev1.Node) error {
 					if len(testMachineSets) != 1 {
 						return errors.New("it should have created one machine set")
 					}
@@ -1603,10 +1610,11 @@ var _ = Describe("machineDeployment", func() {
 				func(testMachineDeployment *machinev1.MachineDeployment, _ *machinev1.MachineSet) {
 					testMachineDeployment.Spec.Selector = &metav1.LabelSelector{}
 				},
-				func(_ *machinev1.MachineDeployment, testMachineSets []machinev1.MachineSet, _ *corev1.Node) error {
+				func(_ *machinev1.MachineDeployment, testMachineSets []machinev1.MachineSet, _ []machinev1.Machine, _ *corev1.Node) error {
 					if len(testMachineSets) > 1 {
 						return errors.New("it should not have created one machine set")
 					}
+
 					return nil
 				},
 			),
@@ -1614,7 +1622,7 @@ var _ = Describe("machineDeployment", func() {
 				func(testMachineDeployment *machinev1.MachineDeployment, _ *machinev1.MachineSet) {
 					testMachineDeployment.DeletionTimestamp = &metav1.Time{Time: time.Now()}
 				},
-				func(testMachineDeployment *machinev1.MachineDeployment, _ []machinev1.MachineSet, _ *corev1.Node) error {
+				func(testMachineDeployment *machinev1.MachineDeployment, _ []machinev1.MachineSet, _ []machinev1.Machine, _ *corev1.Node) error {
 					if len(testMachineDeployment.Finalizers) > 0 {
 						return errors.New("it should have removed the finalizers")
 					}
@@ -1631,7 +1639,7 @@ var _ = Describe("machineDeployment", func() {
 					testMachineSet.Spec = machinev1.MachineSetSpec{}
 					testMachineSet.Status = machinev1.MachineSetStatus{}
 				},
-				func(_ *machinev1.MachineDeployment, testMachineSets []machinev1.MachineSet, _ *corev1.Node) error {
+				func(_ *machinev1.MachineDeployment, testMachineSets []machinev1.MachineSet, _ []machinev1.Machine, _ *corev1.Node) error {
 					if len(testMachineSets) != 0 {
 						return errors.New("it shouldn't have created machine set")
 					}
@@ -1648,7 +1656,7 @@ var _ = Describe("machineDeployment", func() {
 					testMachineSet.Spec = machinev1.MachineSetSpec{}
 					testMachineSet.Status = machinev1.MachineSetStatus{}
 				},
-				func(_ *machinev1.MachineDeployment, testMachineSets []machinev1.MachineSet, _ *corev1.Node) error {
+				func(_ *machinev1.MachineDeployment, testMachineSets []machinev1.MachineSet, _ []machinev1.Machine, _ *corev1.Node) error {
 					if len(testMachineSets) < 1 {
 						return errors.New("it should have created one machine set")
 					}
@@ -1662,7 +1670,7 @@ var _ = Describe("machineDeployment", func() {
 					}
 					testMachineDeployment.Spec.Template.Spec.Class.Name = "MachineClass-test-new"
 				},
-				func(_ *machinev1.MachineDeployment, testMachineSets []machinev1.MachineSet, _ *corev1.Node) error {
+				func(_ *machinev1.MachineDeployment, testMachineSets []machinev1.MachineSet, _ []machinev1.Machine, _ *corev1.Node) error {
 					// Old machine set should exist and should be scaled-down to zero.
 					if len(testMachineSets) == 0 || testMachineSets[0].Spec.Replicas != 0 {
 						return errors.New("it should have scaled-down old machineSet to zero")
@@ -1691,7 +1699,7 @@ var _ = Describe("machineDeployment", func() {
 					}
 					testMachine = &machinev1.Machine{}
 				},
-				func(testMachineDeployment *machinev1.MachineDeployment, testMachineSets []machinev1.MachineSet, _ *corev1.Node) error {
+				func(testMachineDeployment *machinev1.MachineDeployment, testMachineSets []machinev1.MachineSet, _ []machinev1.Machine, _ *corev1.Node) error {
 					if len(testMachineSets) != 2 || testMachineSets[0].Spec.Replicas != testMachineDeployment.Spec.Replicas {
 						return errors.New("it should have fully scaled-up the new machineSet")
 					}
@@ -1704,7 +1712,7 @@ var _ = Describe("machineDeployment", func() {
 					// This should trigger rolling-update.
 					testMachineDeployment.Spec.Template.Spec.Class.Name = "MachineClass-test-new"
 				},
-				func(_ *machinev1.MachineDeployment, testMachineSets []machinev1.MachineSet, _ *corev1.Node) error {
+				func(_ *machinev1.MachineDeployment, testMachineSets []machinev1.MachineSet, _ []machinev1.Machine, _ *corev1.Node) error {
 					if len(testMachineSets) != 2 {
 						return errors.New("it should have created a new machine set")
 					}
@@ -1716,7 +1724,7 @@ var _ = Describe("machineDeployment", func() {
 					// This should trigger rolling-update.
 					testMachineDeployment.Spec.Template.Spec.Class.Name = "MachineClass-test-new"
 				},
-				func(testMachineDeployment *machinev1.MachineDeployment, testMachineSets []machinev1.MachineSet, _ *corev1.Node) error {
+				func(testMachineDeployment *machinev1.MachineDeployment, testMachineSets []machinev1.MachineSet, _ []machinev1.Machine, _ *corev1.Node) error {
 					/*TODO: when we create a new RS then we assign it the replica count and
 					then we again see if the replica count can be increased in the same reconciliation loop
 					So in our case we first assign it replica count of 1(note a new machine is not created)
@@ -1748,7 +1756,7 @@ var _ = Describe("machineDeployment", func() {
 					}
 
 				},
-				func(testMachineDeployment *machinev1.MachineDeployment, _ []machinev1.MachineSet, _ *corev1.Node) error {
+				func(testMachineDeployment *machinev1.MachineDeployment, _ []machinev1.MachineSet, _ []machinev1.Machine, _ *corev1.Node) error {
 					// RollbackTo should be removed after rollback.
 					if testMachineDeployment.Spec.RollbackTo != nil {
 						return errors.New("RollbackTo field should have been removed from machine-deployment")
@@ -1768,7 +1776,7 @@ var _ = Describe("machineDeployment", func() {
 					testMachineDeployment.Spec.Template.Spec.Class.Name = "MachineClass-test-new"
 					testMachineDeployment.Spec.Paused = true
 				},
-				func(_ *machinev1.MachineDeployment, testMachineSets []machinev1.MachineSet, _ *corev1.Node) error {
+				func(_ *machinev1.MachineDeployment, testMachineSets []machinev1.MachineSet, _ []machinev1.Machine, _ *corev1.Node) error {
 
 					if len(testMachineSets) != 1 {
 						return errors.New("there should be only one old machine-set")
@@ -1781,6 +1789,17 @@ var _ = Describe("machineDeployment", func() {
 					return nil
 				},
 			),
+			Entry("should set MachinePriority=1 for the machines named in TriggerDeletionByMCM annotation in the MachineDeployment",
+				func(testMachineDeployment *machinev1.MachineDeployment, _ *machinev1.MachineSet) {
+					testMachineDeployment.Annotations[machineutils.TriggerDeletionByMCM] = annotations.CreateMachinesTriggeredForDeletionAnnotValue([]string{testMachine.Name})
+				},
+				func(_ *machinev1.MachineDeployment, _ []machinev1.MachineSet, machines []machinev1.Machine, _ *corev1.Node) error {
+					if machines[0].Annotations[machineutils.MachinePriority] != "1" {
+						return fmt.Errorf("testMachine should have its %q annotation set to 1", machineutils.MachinePriority)
+					}
+					return nil
+				},
+			),
 		)
 	})
 
diff --git a/pkg/controller/deployment_util.go b/pkg/controller/deployment_util.go
index 130527697f..974f18a02d 100644
--- a/pkg/controller/deployment_util.go
+++ b/pkg/controller/deployment_util.go
@@ -25,6 +25,7 @@ package controller
 import (
 	"context"
 	"fmt"
+	"github.com/gardener/machine-controller-manager/pkg/util/provider/machineutils"
 	"reflect"
 	"sort"
 	"strconv"
@@ -921,7 +922,7 @@ func WaitForMachinesHashPopulated(c v1alpha1listers.MachineSetLister, desiredGen
 func LabelMachinesWithHash(ctx context.Context, machineList *v1alpha1.MachineList, c v1alpha1client.MachineV1alpha1Interface, machineLister v1alpha1listers.MachineLister, namespace, name, hash string) error {
 	for _, machine := range machineList.Items {
 		// Ignore inactive Machines.
-		if !IsMachineActive(&machine) {
+		if !machineutils.IsMachineActive(&machine) {
 			continue
 		}
 		// Only label the machine that doesn't already have the new hash
diff --git a/pkg/controller/machineset.go b/pkg/controller/machineset.go
index 4176406e50..4cbe27fe05 100644
--- a/pkg/controller/machineset.go
+++ b/pkg/controller/machineset.go
@@ -26,6 +26,7 @@ import (
 	"context"
 	"errors"
 	"fmt"
+	"github.com/gardener/machine-controller-manager/pkg/util/provider/machineutils"
 	"reflect"
 	"sort"
 	"sync"
@@ -337,11 +338,11 @@ func (c *controller) manageReplicas(ctx context.Context, allMachines []*v1alpha1
 	}
 
 	var activeMachines, staleMachines []*v1alpha1.Machine
-	for _, machine := range allMachines {
-		if IsMachineActive(machine) {
-			activeMachines = append(activeMachines, machine)
-		} else if IsMachineFailed(machine) {
-			staleMachines = append(staleMachines, machine)
+	for _, m := range allMachines {
+		if machineutils.IsMachineFailed(m) || machineutils.IsMachineTriggeredForDeletion(m) {
+			staleMachines = append(staleMachines, m)
+		} else if machineutils.IsMachineActive(m) {
+			activeMachines = append(activeMachines, m)
 		}
 	}
 
@@ -685,7 +686,7 @@ func (c *controller) prepareMachineForDeletion(ctx context.Context, targetMachin
 	} else {
 		// successful delete of a Failed phase machine due to unhealthiness for too long, increments staleMachinesRemoved counter
 		// note: call is blocking and thread safe as other worker threads might be updating the counter as well
-		if IsMachineFailed(targetMachine) && targetMachine.Status.LastOperation.Type == v1alpha1.MachineOperationHealthCheck {
+		if machineutils.IsMachineFailed(targetMachine) && targetMachine.Status.LastOperation.Type == v1alpha1.MachineOperationHealthCheck {
 			staleMachinesRemoved.increment()
 		}
 	}
diff --git a/pkg/controller/machineset_test.go b/pkg/controller/machineset_test.go
index d3a85e1784..a8d9a2ff94 100644
--- a/pkg/controller/machineset_test.go
+++ b/pkg/controller/machineset_test.go
@@ -7,6 +7,7 @@ package controller
 import (
 	"context"
 	"errors"
+	"github.com/gardener/machine-controller-manager/pkg/util/provider/machineutils"
 	"sync"
 	"time"
 
@@ -608,6 +609,7 @@ var _ = Describe("machineset", func() {
 					Labels: map[string]string{
 						"test-label": "test-label",
 					},
+					Annotations: map[string]string{},
 				},
 				TypeMeta: metav1.TypeMeta{
 					Kind:       "Machine",
@@ -639,6 +641,46 @@ var _ = Describe("machineset", func() {
 					},
 				},
 			}
+
+			testActiveMachine3 = &machinev1.Machine{
+				ObjectMeta: metav1.ObjectMeta{
+					Name:      "machine-3",
+					Namespace: testNamespace,
+					UID:       "12345610",
+					Labels: map[string]string{
+						"test-label": "test-label",
+					},
+				},
+				TypeMeta: metav1.TypeMeta{
+					Kind:       "Machine",
+					APIVersion: "machine.sapcloud.io/v1alpha1",
+				},
+				Status: machinev1.MachineStatus{
+					CurrentStatus: machinev1.CurrentStatus{
+						Phase: MachineRunning,
+					},
+				},
+			}
+
+			testActiveMachine4 = &machinev1.Machine{
+				ObjectMeta: metav1.ObjectMeta{
+					Name:      "machine-4",
+					Namespace: testNamespace,
+					UID:       "12345611",
+					Labels: map[string]string{
+						"test-label": "test-label",
+					},
+				},
+				TypeMeta: metav1.TypeMeta{
+					Kind:       "Machine",
+					APIVersion: "machine.sapcloud.io/v1alpha1",
+				},
+				Status: machinev1.MachineStatus{
+					CurrentStatus: machinev1.CurrentStatus{
+						Phase: MachineFailed,
+					},
+				},
+			}
 		})
 
 		// Testcase: ActiveMachines < DesiredMachines
@@ -716,26 +758,6 @@ var _ = Describe("machineset", func() {
 			stop := make(chan struct{})
 			defer close(stop)
 
-			testActiveMachine3 = &machinev1.Machine{
-				ObjectMeta: metav1.ObjectMeta{
-					Name:      "machine-3",
-					Namespace: testNamespace,
-					UID:       "12345610",
-					Labels: map[string]string{
-						"test-label": "test-label",
-					},
-				},
-				TypeMeta: metav1.TypeMeta{
-					Kind:       "Machine",
-					APIVersion: "machine.sapcloud.io/v1alpha1",
-				},
-				Status: machinev1.MachineStatus{
-					CurrentStatus: machinev1.CurrentStatus{
-						Phase: MachineRunning,
-					},
-				},
-			}
-
 			objects := []runtime.Object{}
 			objects = append(objects, testMachineSet, testActiveMachine1, testActiveMachine2, testActiveMachine3)
 			c, trackers := createController(stop, testNamespace, objects, nil, nil)
@@ -779,26 +801,6 @@ var _ = Describe("machineset", func() {
 				},
 			}
 
-			testActiveMachine4 = &machinev1.Machine{
-				ObjectMeta: metav1.ObjectMeta{
-					Name:      "machine-4",
-					Namespace: testNamespace,
-					UID:       "12345611",
-					Labels: map[string]string{
-						"test-label": "test-label",
-					},
-				},
-				TypeMeta: metav1.TypeMeta{
-					Kind:       "Machine",
-					APIVersion: "machine.sapcloud.io/v1alpha1",
-				},
-				Status: machinev1.MachineStatus{
-					CurrentStatus: machinev1.CurrentStatus{
-						Phase: MachineFailed,
-					},
-				},
-			}
-
 			objects := []runtime.Object{}
 			objects = append(objects, testMachineSet, testActiveMachine1, testActiveMachine2, testActiveMachine3, testActiveMachine4)
 			c, trackers := createController(stop, testNamespace, objects, nil, nil)
@@ -822,46 +824,6 @@ var _ = Describe("machineset", func() {
 			stop := make(chan struct{})
 			defer close(stop)
 
-			testActiveMachine3 = &machinev1.Machine{
-				ObjectMeta: metav1.ObjectMeta{
-					Name:      "machine-3",
-					Namespace: testNamespace,
-					UID:       "12345610",
-					Labels: map[string]string{
-						"test-label": "test-label",
-					},
-				},
-				TypeMeta: metav1.TypeMeta{
-					Kind:       "Machine",
-					APIVersion: "machine.sapcloud.io/v1alpha1",
-				},
-				Status: machinev1.MachineStatus{
-					CurrentStatus: machinev1.CurrentStatus{
-						Phase: MachineRunning,
-					},
-				},
-			}
-
-			testActiveMachine4 = &machinev1.Machine{
-				ObjectMeta: metav1.ObjectMeta{
-					Name:      "machine-4",
-					Namespace: testNamespace,
-					UID:       "12345611",
-					Labels: map[string]string{
-						"test-label": "test-label",
-					},
-				},
-				TypeMeta: metav1.TypeMeta{
-					Kind:       "Machine",
-					APIVersion: "machine.sapcloud.io/v1alpha1",
-				},
-				Status: machinev1.MachineStatus{
-					CurrentStatus: machinev1.CurrentStatus{
-						Phase: MachineRunning,
-					},
-				},
-			}
-
 			objects := []runtime.Object{}
 			objects = append(objects, testMachineSet, testActiveMachine1, testActiveMachine2, testActiveMachine3, testActiveMachine4)
 			c, trackers := createController(stop, testNamespace, objects, nil, nil)
@@ -879,6 +841,38 @@ var _ = Describe("machineset", func() {
 			Expect(Err).Should(BeNil())
 		})
 
+		It("should delete MachinePriority=1 machines and spawn replacement machine", func() {
+			stop := make(chan struct{})
+			defer close(stop)
+			objects := []runtime.Object{}
+
+			staleMachine := testActiveMachine1.DeepCopy()
+			staleMachine.Annotations[machineutils.MachinePriority] = "1"
+
+			objects = append(objects, testMachineSet, staleMachine, testActiveMachine2, testActiveMachine3)
+			c, trackers := createController(stop, testNamespace, objects, nil, nil)
+			defer trackers.Stop()
+			waitForCacheSync(stop, c)
+
+			machines, _ := c.controlMachineClient.Machines(testNamespace).List(context.TODO(), metav1.ListOptions{})
+			Expect(len(machines.Items)).To(Equal(int(testMachineSet.Spec.Replicas)))
+
+			beforeMachines := []*machinev1.Machine{staleMachine, testActiveMachine2, testActiveMachine3}
+			err := c.manageReplicas(context.TODO(), beforeMachines, testMachineSet)
+			Expect(err).Should(BeNil())
+			waitForCacheSync(stop, c)
+
+			_, err = c.controlMachineClient.Machines(testNamespace).Get(context.Background(), staleMachine.Name, metav1.GetOptions{})
+			Expect(err).ShouldNot(BeNil())
+			Expect(err).To(Satisfy(func(e error) bool {
+				return k8sError.IsNotFound(e)
+			}))
+			afterMachines, err := c.controlMachineClient.Machines(testNamespace).List(context.TODO(), metav1.ListOptions{})
+			// replica count is still maintained.
+			Expect(len(afterMachines.Items)).To(Equal(int(testMachineSet.Spec.Replicas)))
+			Expect(err).Should(BeNil())
+
+		})
 	})
 
 	// TODO: This method has dependency on generic-machineclass. Implement later.
diff --git a/pkg/controller/machineset_util.go b/pkg/controller/machineset_util.go
index 9227af4405..69d8ca6c45 100644
--- a/pkg/controller/machineset_util.go
+++ b/pkg/controller/machineset_util.go
@@ -89,7 +89,7 @@ func (c *controller) syncMachinesNodeTemplates(ctx context.Context, machineList
 
 	for _, machine := range machineList {
 		// Ignore inactive Machines.
-		if !IsMachineActive(machine) {
+		if !machineutils.IsMachineActive(machine) {
 			continue
 		}
 
@@ -155,7 +155,7 @@ func (c *controller) syncMachinesConfig(ctx context.Context, machineList []*v1al
 
 	for _, machine := range machineList {
 		// Ignore inactive Machines.
-		if !IsMachineActive(machine) {
+		if !machineutils.IsMachineActive(machine) {
 			continue
 		}
 
diff --git a/pkg/util/provider/machineutils/utils.go b/pkg/util/provider/machineutils/utils.go
index af1458bffe..7d86d41765 100644
--- a/pkg/util/provider/machineutils/utils.go
+++ b/pkg/util/provider/machineutils/utils.go
@@ -50,9 +50,9 @@ const (
 	// NotManagedByMCM annotation helps in identifying the nodes which are not handled by MCM
 	NotManagedByMCM = "node.machine.sapcloud.io/not-managed-by-mcm"
 
-	// TriggerDeletionByMCM annotation on the node would trigger the deletion of the corresponding machine object in the control cluster
+	// TriggerDeletionByMCM annotation on the node or machine would trigger the deletion of the corresponding node and machine object in the control cluster
 	// This annotation can also be set on the MachineDeployment and contains the machine names for which deletion should be triggered.
-	// This feature is leveraged by the CA-MCM cloud provider.
+	// The latter feature is leveraged by the CA-MCM cloud provider.
 	TriggerDeletionByMCM = "node.machine.sapcloud.io/trigger-deletion-by-mcm"
 
 	// NodeUnhealthy is a node termination reason for failed machines
@@ -101,3 +101,18 @@ func IsMachineFailedOrTerminating(machine *v1alpha1.Machine) bool {
 	}
 	return false
 }
+
+// IsMachineActive checks if machine was active
+func IsMachineActive(p *v1alpha1.Machine) bool {
+	return p.Status.CurrentStatus.Phase != v1alpha1.MachineFailed && p.Status.CurrentStatus.Phase != v1alpha1.MachineTerminating
+}
+
+// IsMachineFailed checks if machine has failed
+func IsMachineFailed(p *v1alpha1.Machine) bool {
+	return p.Status.CurrentStatus.Phase == v1alpha1.MachineFailed
+}
+
+// IsMachineTriggeredForDeletion checks if machine was triggered for deletion
+func IsMachineTriggeredForDeletion(m *v1alpha1.Machine) bool {
+	return m.Annotations[MachinePriority] == "1" || m.Annotations[TriggerDeletionByMCM] == "true"
+}


Reviews:
user: login: aaronfern
body: Thanks for the PR!
I just have one question to clarify
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-16T06:24:31Z
user: login: elankath
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-17T11:00:13Z
user: login: takoverflow
state: APPROVED
author_association: MEMBER
submitted_at: 2025-03-20T07:42:08Z
user: login: unmarshall
body: Minor comments.
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-20T08:17:16Z
user: login: elankath
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-20T08:31:18Z
user: login: elankath
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-20T08:41:05Z
user: login: aaronfern
state: APPROVED
author_association: MEMBER
submitted_at: 2025-03-21T05:45:07Z

Review comments:
diff_hunk: @@ -101,3 +101,23 @@ func IsMachineFailedOrTerminating(machine *v1alpha1.Machine) bool {
 	}
 	return false
 }
+
+// IsMachineActive checks if machine was active
+func IsMachineActive(p *v1alpha1.Machine) bool {
+	if p.Status.CurrentStatus.Phase == v1alpha1.MachineFailed {
+		return false
+	} else if p.Status.CurrentStatus.Phase == v1alpha1.MachineTerminating {
+		return false
+	}
+	return true
+}
+
+// IsMachineFailed checks if machine has failed
+func IsMachineFailed(p *v1alpha1.Machine) bool {
+	return p.Status.CurrentStatus.Phase == v1alpha1.MachineFailed
+}
+
+// IsMachineTriggeredForDeletion checks if machine was triggered for deletion
+func IsMachineTriggeredForDeletion(m *v1alpha1.Machine) bool {
+	return m.Annotations[MachinePriority] == "1" || m.Annotations[TriggerDeletionByMCM] == "true"
path: pkg/util/provider/machineutils/utils.go
user: login: aaronfern
body: I wondered why do we need to check for `m.Annotations[TriggerDeletionByMCM] == "true"`?
This annotation is either added to the `mcd` (by CA) or to a node (by user)
I don't see any code that adds this annotation to the machine unless I missed something 
created_at: 2025-03-16T06:23:02Z
author_association: MEMBER
diff_hunk: @@ -101,3 +101,23 @@ func IsMachineFailedOrTerminating(machine *v1alpha1.Machine) bool {
 	}
 	return false
 }
+
+// IsMachineActive checks if machine was active
+func IsMachineActive(p *v1alpha1.Machine) bool {
+	if p.Status.CurrentStatus.Phase == v1alpha1.MachineFailed {
+		return false
+	} else if p.Status.CurrentStatus.Phase == v1alpha1.MachineTerminating {
+		return false
+	}
+	return true
+}
+
+// IsMachineFailed checks if machine has failed
+func IsMachineFailed(p *v1alpha1.Machine) bool {
+	return p.Status.CurrentStatus.Phase == v1alpha1.MachineFailed
+}
+
+// IsMachineTriggeredForDeletion checks if machine was triggered for deletion
+func IsMachineTriggeredForDeletion(m *v1alpha1.Machine) bool {
+	return m.Annotations[MachinePriority] == "1" || m.Annotations[TriggerDeletionByMCM] == "true"
path: pkg/util/provider/machineutils/utils.go
user: login: elankath
body: This annotation was added to the `Node` object to trigger for deletion. So I thought for symmetry it should also be added to `Machine` object. 
created_at: 2025-03-17T11:00:13Z
author_association: MEMBER
diff_hunk: @@ -101,3 +101,23 @@ func IsMachineFailedOrTerminating(machine *v1alpha1.Machine) bool {
 	}
 	return false
 }
+
+// IsMachineActive checks if machine was active
+func IsMachineActive(p *v1alpha1.Machine) bool {
path: pkg/util/provider/machineutils/utils.go
user: login: unmarshall
body: you can simplify it:
```go
return ! (p.Status.CurrentPhase.Phase == v1alpha1.MachineFailed || p.Status.CurrentStatus.Phase == v1alpha1.MachineTerminating)

```
created_at: 2025-03-20T08:05:59Z
author_association: MEMBER
diff_hunk: @@ -101,3 +101,23 @@ func IsMachineFailedOrTerminating(machine *v1alpha1.Machine) bool {
 	}
 	return false
 }
+
+// IsMachineActive checks if machine was active
+func IsMachineActive(p *v1alpha1.Machine) bool {
+	if p.Status.CurrentStatus.Phase == v1alpha1.MachineFailed {
+		return false
+	} else if p.Status.CurrentStatus.Phase == v1alpha1.MachineTerminating {
+		return false
+	}
+	return true
+}
+
+// IsMachineFailed checks if machine has failed
+func IsMachineFailed(p *v1alpha1.Machine) bool {
+	return p.Status.CurrentStatus.Phase == v1alpha1.MachineFailed
+}
+
+// IsMachineTriggeredForDeletion checks if machine was triggered for deletion
+func IsMachineTriggeredForDeletion(m *v1alpha1.Machine) bool {
+	return m.Annotations[MachinePriority] == "1" || m.Annotations[TriggerDeletionByMCM] == "true"
path: pkg/util/provider/machineutils/utils.go
user: login: unmarshall
body: Current doc string:
```go
// TriggerDeletionByMCM annotation on the node would trigger the deletion of the corresponding machine object in the control cluster
// This annotation can also be set on the MachineDeployment and contains the machine names for which deletion should be triggered.
// This feature is leveraged by the CA-MCM cloud provider.
TriggerDeletionByMCM = "node.machine.sapcloud.io/trigger-deletion-by-mcm"
```
As per the doc string the annotation can be placed on MCD or on the Node. Apart from symmetry is there any reason to place this label on the `Machine`?
created_at: 2025-03-20T08:10:09Z
author_association: MEMBER
diff_hunk: @@ -879,6 +841,40 @@ var _ = Describe("machineset", func() {
 			Expect(Err).Should(BeNil())
 		})
 
+		It("should delete MachinePriority=1 machines and spawn replacement machine", func() {
+			stop := make(chan struct{})
+			defer close(stop)
+			objects := []runtime.Object{}
+
+			staleMachine := testActiveMachine1.DeepCopy()
+			staleMachine.Annotations[machineutils.MachinePriority] = "1"
+
+			objects = append(objects, testMachineSet, staleMachine, testActiveMachine2, testActiveMachine3)
+			c, trackers := createController(stop, testNamespace, objects, nil, nil)
+			defer trackers.Stop()
+			waitForCacheSync(stop, c)
+
+			machines, _ := c.controlMachineClient.Machines(testNamespace).List(context.TODO(), metav1.ListOptions{})
+			Expect(len(machines.Items)).To(Equal(int(testMachineSet.Spec.Replicas)))
+
+			beforeMachines := []*machinev1.Machine{staleMachine, testActiveMachine2, testActiveMachine3}
+			err := c.manageReplicas(context.TODO(), beforeMachines, testMachineSet)
+			Expect(err).Should(BeNil())
+			waitForCacheSync(stop, c)
+
+			_, err = c.controlMachineClient.Machines(testNamespace).Get(context.Background(), staleMachine.Name, metav1.GetOptions{})
+			// MachinePriority=1 staleMachine is no longer found
+			//Expect(staleMachine).Should(BeNil())
path: pkg/controller/machineset_test.go
user: login: unmarshall
body: can you remove this commented code line?
created_at: 2025-03-20T08:11:23Z
author_association: MEMBER
diff_hunk: @@ -879,6 +841,40 @@ var _ = Describe("machineset", func() {
 			Expect(Err).Should(BeNil())
 		})
 
+		It("should delete MachinePriority=1 machines and spawn replacement machine", func() {
+			stop := make(chan struct{})
+			defer close(stop)
+			objects := []runtime.Object{}
+
+			staleMachine := testActiveMachine1.DeepCopy()
+			staleMachine.Annotations[machineutils.MachinePriority] = "1"
+
+			objects = append(objects, testMachineSet, staleMachine, testActiveMachine2, testActiveMachine3)
+			c, trackers := createController(stop, testNamespace, objects, nil, nil)
+			defer trackers.Stop()
+			waitForCacheSync(stop, c)
+
+			machines, _ := c.controlMachineClient.Machines(testNamespace).List(context.TODO(), metav1.ListOptions{})
+			Expect(len(machines.Items)).To(Equal(int(testMachineSet.Spec.Replicas)))
+
+			beforeMachines := []*machinev1.Machine{staleMachine, testActiveMachine2, testActiveMachine3}
+			err := c.manageReplicas(context.TODO(), beforeMachines, testMachineSet)
+			Expect(err).Should(BeNil())
+			waitForCacheSync(stop, c)
+
+			_, err = c.controlMachineClient.Machines(testNamespace).Get(context.Background(), staleMachine.Name, metav1.GetOptions{})
+			// MachinePriority=1 staleMachine is no longer found
+			//Expect(staleMachine).Should(BeNil())
path: pkg/controller/machineset_test.go
user: login: elankath
body: Some times `Node` may not be registered but operator wants MCM to trigger deletion before the machine timeout. This will assist in that case. 
created_at: 2025-03-20T08:31:18Z
author_association: MEMBER
diff_hunk: @@ -101,3 +101,23 @@ func IsMachineFailedOrTerminating(machine *v1alpha1.Machine) bool {
 	}
 	return false
 }
+
+// IsMachineActive checks if machine was active
+func IsMachineActive(p *v1alpha1.Machine) bool {
+	if p.Status.CurrentStatus.Phase == v1alpha1.MachineFailed {
+		return false
+	} else if p.Status.CurrentStatus.Phase == v1alpha1.MachineTerminating {
+		return false
+	}
+	return true
+}
+
+// IsMachineFailed checks if machine has failed
+func IsMachineFailed(p *v1alpha1.Machine) bool {
+	return p.Status.CurrentStatus.Phase == v1alpha1.MachineFailed
+}
+
+// IsMachineTriggeredForDeletion checks if machine was triggered for deletion
+func IsMachineTriggeredForDeletion(m *v1alpha1.Machine) bool {
+	return m.Annotations[MachinePriority] == "1" || m.Annotations[TriggerDeletionByMCM] == "true"
path: pkg/util/provider/machineutils/utils.go
user: login: elankath
body: corrected godoc and removed commented code
created_at: 2025-03-20T08:41:05Z
author_association: MEMBER

PR comments:
user: login: CLAassistant
created_at: 2025-03-10T07:40:58Z
body: [![CLA assistant check](https://cla-assistant.io/pull/badge/not_signed)](https://cla-assistant.io/gardener/machine-controller-manager?pullRequest=972) <br/>Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our [Contributor License Agreement](https://cla-assistant.io/gardener/machine-controller-manager?pullRequest=972) before we can accept your contribution.<br/><sub>You have signed the CLA already but the status is still pending? Let us [recheck](https://cla-assistant.io/check/gardener/machine-controller-manager?pullRequest=972) it.</sub>
author_association: NONE
user: login: elankath
created_at: 2025-03-10T07:59:37Z
body: # Test Log

## Situation Before Fix

### Initial Cluster State

#### List MCD and MC
```
machine-controller-manager ➤ k get mcd; k get mc #list                                        git:master
NAME                                READY   DESIRED   UP-TO-DATE   AVAILABLE   AGE
shoot--i034796--aw-one-zone-z1                                                 3m32s
shoot--i034796--aw-sys-comp-z1      2       2         2            2           3m32s
shoot--i034796--aw-three-zones-z1   1       1         1            1           3m32s
shoot--i034796--aw-three-zones-z2                                              3m32s
shoot--i034796--aw-three-zones-z3                                              3m32s
NAME                                            STATUS    AGE     NODE
shoot--i034796--aw-sys-comp-z1-6c5b8-g6phv      Running   3m28s   shoot--i034796--aw-sys-comp-z1-6c5b8-g6phv
shoot--i034796--aw-sys-comp-z1-6c5b8-p5sdh      Running   3m27s   shoot--i034796--aw-sys-comp-z1-6c5b8-p5sdh
shoot--i034796--aw-three-zones-z1-6547d-wt5qw   Running   3m28s   shoot--i034796--aw-three-zones-z1-6547d-wt5qw
```

### Annotate Machine Like CA Does but don't change replicas

```
k annotate mcd shoot--i034796--aw-three-zones-z1 "node.machine.sapcloud.io/trigger-deletion-by-mcm=shoot--i034796--aw-three-zones-z1-6547d-wt5qw"
machinedeployment.machine.sapcloud.io/shoot--i034796--aw-three-zones-z1 annotated
```

### Log acknowledges this but MachineSet controller doesn't react

```
I0310 07:51:39.098225  141546 deployment.go:435] Processing the machinedeployment "shoot--i034796--aw-three-zones-z1" (with replicas 1)
I0310 07:51:39.098305  141546 deployment.go:621] MachineDeployment "shoot--i034796--aw-three-zones-z1" has #1 machine(s) marked for deletion, triggerForDeletionMachineNames=[shoot--i034796--aw-three-zones-z1-6547d-wt5qw]
```

### Explicitly Reduce MCD replicas by 1

```
k scale mcd shoot--i034796--aw-three-zones-z1 --replicas=0       git:master
machinedeployment.machine.sapcloud.io/shoot--i034796--aw-three-zones-z1 scaled
```

### MachineSet controller now acts as expected and removes MC

#### Log
```
I0310 07:53:37.850483  141546 machineset.go:500] Processing the machineset "shoot--i034796--aw-three-zones-z1-6547d" with replicas 0 associated with machine class: ""
I0310 07:53:37.850644  141546 machineset.go:427] Too many replicas for  shoot--i034796--aw/shoot--i034796--aw-three-zones-z1-6547d, need 0, deleting 1
I0310 07:53:37.850675  141546 machineset_util.go:210] Machine "shoot--i034796--aw-three-zones-z1-6547d-wt5qw" has machinepriority.machine.sapcloud.io annotation set to 1
I0310 07:53:37.850683  141546 machineset_util.go:217] Machine "shoot--i034796--aw-three-zones-z1-6547d-wt5qw" needs to be deleted
```
#### Status
```
k get mcd; k get mc #list                                        git:master
NAME                                READY   DESIRED   UP-TO-DATE   AVAILABLE   AGE
shoot--i034796--aw-one-zone-z1                                                 7m44s
shoot--i034796--aw-sys-comp-z1      2       2         2            2           7m44s
shoot--i034796--aw-three-zones-z1                                              7m44s
shoot--i034796--aw-three-zones-z2                                              7m44s
shoot--i034796--aw-three-zones-z3                                              7m44s
NAME                                         STATUS    AGE     NODE
shoot--i034796--aw-sys-comp-z1-6c5b8-g6phv   Running   7m40s   shoot--i034796--aw-sys-comp-z1-6c5b8-g6phv
shoot--i034796--aw-sys-comp-z1-6c5b8-p5sdh   Running   7m39s   shoot--i034796--aw-sys-comp-z1-6c5b8-p5sdh
```

## Situation After Fix

Initially we had a really complicated fix that adjusted both CA and MCM to take into account the situation where CA "changes its mind" about deleting the machine. But this was too complicated and ran into several issues and so we took brain-dead, simple fix in MCM where a machine tainted by the CA is marked as stale by the machine set controller. This solves the problem as machine is now deleted ASAP and kube-proxy will not get the chance to report it as unhealthy.

### Initial Cluster State

```
machine-controller-manager ➤ k get mcd; k get mc #list                                       git:kp-fix*
NAME                                READY   DESIRED   UP-TO-DATE   AVAILABLE   AGE
shoot--i034796--aw-one-zone-z1                                                 57m
shoot--i034796--aw-sys-comp-z1      2       2         2            2           57m
shoot--i034796--aw-three-zones-z1   1       1         1            1           57m
shoot--i034796--aw-three-zones-z2                                              57m
shoot--i034796--aw-three-zones-z3                                              57m
NAME                                            STATUS    AGE     NODE
shoot--i034796--aw-sys-comp-z1-6c5b8-jq79g      Running   3m47s   shoot--i034796--aw-sys-comp-z1-6c5b8-jq79g
shoot--i034796--aw-sys-comp-z1-6c5b8-sqzzr      Running   3m48s   shoot--i034796--aw-sys-comp-z1-6c5b8-sqzzr
shoot--i034796--aw-three-zones-z1-6547d-lbc5c   Running   2m18s   shoot--i034796--aw-three-zones-z1-6547d-lbc5c
```

### Simulate CA tainting

```
k annotate mcd shoot--i034796--aw-three-zones-z1 "node.machine.sapcloud.io/trigger-deletion-by-mcm=shoot--i034796--aw-three-zones-z1-6547d-lbc5c"
```

### Machine is moved immediately to Terminating without waiting for replica change


```
k get mcd; k get mc #list                                       git:kp-fix*
NAME                                READY   DESIRED   UP-TO-DATE   AVAILABLE   AGE
shoot--i034796--aw-one-zone-z1                                                 59m
shoot--i034796--aw-sys-comp-z1      2       2         2            2           59m
shoot--i034796--aw-three-zones-z1           1         2                        59m
shoot--i034796--aw-three-zones-z2                                              59m
shoot--i034796--aw-three-zones-z3                                              59m
NAME                                            STATUS        AGE     NODE
shoot--i034796--aw-sys-comp-z1-6c5b8-jq79g      Running       5m      shoot--i034796--aw-sys-comp-z1-6c5b8-jq79g
shoot--i034796--aw-sys-comp-z1-6c5b8-sqzzr      Running       5m1s    shoot--i034796--aw-sys-comp-z1-6c5b8-sqzzr
shoot--i034796--aw-three-zones-z1-6547d-lbc5c   Terminating   3m31s   shoot--i034796--aw-three-zones-z1-6547d-lbc5c
shoot--i034796--aw-three-zones-z1-6547d-sfj9g                 2s
```

### Final Cluster State

```
k get mc                                                        git:kp-fix*
NAME                                            STATUS    AGE     NODE
shoot--i034796--aw-sys-comp-z1-6c5b8-jq79g      Running   5m30s   shoot--i034796--aw-sys-comp-z1-6c5b8-jq79g
shoot--i034796--aw-sys-comp-z1-6c5b8-sqzzr      Running   5m31s   shoot--i034796--aw-sys-comp-z1-6c5b8-sqzzr
shoot--i034796--aw-three-zones-z1-6547d-sfj9g   Running   32s     shoot--i034796--aw-three-zones-z1-6547d-sfj9g
```

### Relevant MCM Log


```
I0310 07:32:34.427722  139138 deployment.go:621] MachineDeployment "shoot--i034796--aw-three-zones-z1" has #1 machine(s) marked for deletion, triggerForDeletionMachineNames=[shoot--i034796--aw-three-zones-z1-6547d-lbc5c]
I0310 07:32:34.435595  139138 machineset.go:350] Deleting stale machines [shoot--i034796--aw-three-zones-z1-6547d-lbc5c]   

I0310 07:32:34.440094  139138 event.go:377] Event(v1.ObjectReference{Kind:"MachineSet", Namespace:"shoot--i034796--aw", Name:"shoot--i034796--aw-three-zones-z1-6547d", UID:"eea0b137-645e-4bd8-88a3-f7cafd54fc52", APIVersion:"machine.sapcloud.io/v1alpha1", ResourceVersion:"986", FieldPath:""}): type: 'Normal' reason: 'SuccessfulDelete' Deleted machine: shoot--i034796--aw-three-zones-z1-6547d-lbc5c

I0310 07:32:34.456524  139138 deployment.go:669] Updated MachineDeployment "shoot--i034796--aw-three-zones-z1" with #0 machine names still pending deletion, triggerDeletionAnnotValue=""

I0310 07:32:34.460161  139138 machineset.go:710] Delete machine from machineset "shoot--i034796--aw-three-zones-z1-6547d-lbc5c"
I0310 07:32:34.460240  139138 machineset.go:380] Too few replicas for MachineSet shoot--i034796--aw-three-zones-z1-6547d, need 1, creating 1
I0310 07:32:34.462233  139138 deployment.go:435] Processing the machinedeployment "shoot--i034796--aw-three-zones-z1" (with replicas 1)
I0310 07:32:34.466029  139138 controller_utils.go:595] Controller shoot--i034796--aw-three-zones-z1-6547d created machine shoot--i034796--aw-three-zones-z1-6547d-sfj9g

I0310 07:32:34.466261  139138 event.go:377] Event(v1.ObjectReference{Kind:"MachineSet", Namespace:"shoot--i034796--aw", Name:"shoot--i034796--aw-three-zones-z1-6547d", UID:"eea0b137-645e-4bd8-88a3-f7cafd54fc52", APIVersion:"machine.sapcloud.io/v1alpha1", ResourceVersion:"986", FieldPath:""}): type: 'Normal' reason: 'SuccessfulCreate' Created Machine: shoot--i034796--aw-three-zones-z1-6547d-sfj9g

I0310 07:32:34.470251  139138 machineset.go:350] Deleting stale machines [shoot--i034796--aw-three-zones-z1-6547d-lbc5c] 

```

author_association: MEMBER