Repository: gardener/machine-controller-manager

Issue:
Scalability issues with MCM
**How to categorize this issue?**
<!--
Please select area, kind, and priority for this issue. This helps the community categorizing it.
Replace below TODOs or exchange the existing identifiers with those that fit best in your opinion.
If multiple identifiers make sense you can also state the commands multiple times, e.g.
  /area control-plane
  /area auto-scaling
  ...

"/area" identifiers:     audit-logging|auto-scaling|backup|certification|control-plane-migration|control-plane|cost|delivery|dev-productivity|disaster-recovery|documentation|high-availability|logging|metering|monitoring|networking|open-source|ops-productivity|os|performance|quality|robustness|scalability|security|storage|testing|usability|user-management
"/kind" identifiers:     api-change|bug|cleanup|discussion|enhancement|epic|impediment|poc|post-mortem|question|regression|task|technical-debt|test
"/priority" identifiers: 1|2|3|4|5 (ordered from greatest to least)
-->
/area control-plane
/area scalability
/kind enhancement
/priority 1

**What happened**:
In the recent live update, we saw that the worker pools for our seeds were replaced with new ones concurrently. This caused the deletion of old machine-level objects (machine deployments, machine sets, machine classes and machines) and subsequent creation of new machine objects. All the old machines across worker pools went to `Terminating` state simultaneously. 

Since we have a shared queue and around 50 workers picking up items from this queue, this caused massive throttling due to certain potentially long-running operations like the draining of nodes. Because the workers were blocked in the drain operation, the create requests were getting stuck in the queue with no worker available to process these requests. 

The drain timeout was 2 hrs but it took more than 4 hrs because of https://github.com/gardener/machine-controller-manager/issues/785 which is part of `0.54.0` version of MCM which had not reached live landscape with the corresponding mcm-provider release. 

In the recent live update:-
1. 5-6 new worker pools were introduced to replace the existing 2 worker pools.
2. We observed around >100 or so machines in the `Terminating` state. 
3. Around 50 or so create machine requests were stuck in the queue.
4. For around 4 hrs, due to long drain times and throttling, the machines were stuck in drain.


**What you expected to happen**:
MCM should scale much beyond handling 100 concurrent deletion/creation requests. 


Hints from issue comments:
@hoeltcl: Referenced in PTASK0034014 as preventive measure. Do you already have a ETA date?@gardener-robot: @hoeltcl You have mentioned internal references in the public. Please check.@elankath: This is in progress. ETA will be updated later.@hoeltcl: I've extended the due date in the linked problem task to 2025-01-31. Do you think you will be able to make it until that date?@elankath: @hoeltcl we will need more time since we need to test for regressions after separation of machine creation/termination into new work-queues and ensure we properly avoid race conditions. I will update the ticket with revised date.

Solution patch:
diff --git a/.ci/test b/.ci/test
index d13330b9d..a496c4fec 100755
--- a/.ci/test
+++ b/.ci/test
@@ -36,7 +36,7 @@ function test_with_coverage() {
   local output_dir=test/output
   local coverprofile_file=coverprofile.out
   mkdir -p test/output
-  ginkgo $GINKGO_COMMON_FLAGS --coverprofile ${coverprofile_file} -covermode=set -outputdir ${output_dir} ${TEST_PACKAGES}
+  ginkgo $GINKGO_COMMON_FLAGS --coverprofile ${coverprofile_file} -covermode=set -output-dir ${output_dir} ${TEST_PACKAGES}
 
   sed -i -e '/mode: set/d' ${output_dir}/${coverprofile_file}
   {( echo "mode: set"; cat ${output_dir}/${coverprofile_file} )} > ${output_dir}/${coverprofile_file}.temp
@@ -51,7 +51,7 @@ if [[ "${SKIP_UNIT_TESTS}" != "" ]]; then
 else
   echo ">>>>> Invoking unit tests"
   TEST_PACKAGES="cmd pkg"
-  GINKGO_COMMON_FLAGS="-r -timeout=1h0m0s --randomizeAllSpecs --randomizeSuites --failOnPending  --progress"
+  GINKGO_COMMON_FLAGS="-r -timeout=1h0m0s --randomize-all --randomize-suites --fail-on-pending  --show-node-events"
   test_with_coverage
   echo ">>>>> Finished executing unit tests"
 fi
diff --git a/VERSION b/VERSION
index 34ede3a7d..d5323e55b 100644
--- a/VERSION
+++ b/VERSION
@@ -1 +1 @@
-v0.57.0-dev
\ No newline at end of file
+v0.58.0-dev
\ No newline at end of file
diff --git a/docs/FAQ.md b/docs/FAQ.md
index b492b5069..b4bbd6de7 100644
--- a/docs/FAQ.md
+++ b/docs/FAQ.md
@@ -1,6 +1,8 @@
 ---
-title: Machine Controller Manager FAQ
+title: FAQ
 description: Commonly asked questions about MCM
+weight: 5
+persona: Operators
 ---
 
 # Frequently Asked Questions
diff --git a/docs/documents/_index.md b/docs/documents/_index.md
new file mode 100644
index 000000000..bdfec1968
--- /dev/null
+++ b/docs/documents/_index.md
@@ -0,0 +1,5 @@
+---
+title: Documents
+weight: 2
+persona: Developers
+---
\ No newline at end of file
diff --git a/docs/documents/apis.md b/docs/documents/apis.md
index 7e702fd50..cfda6b2e1 100644
--- a/docs/documents/apis.md
+++ b/docs/documents/apis.md
@@ -1055,6 +1055,36 @@ Kubernetes meta/v1.Duration
 </tr>
 <tr>
 <td>
+<code>inPlaceUpdateTimeout</code>
+</td>
+<td>
+<em>
+<a href="https://godoc.org/k8s.io/apimachinery/pkg/apis/meta/v1#Duration">
+Kubernetes meta/v1.Duration
+</a>
+</em>
+</td>
+<td>
+<em>(Optional)</em>
+<p>MachineInPlaceUpdateTimeout is the timeout after which in-place update is declared failed.</p>
+</td>
+</tr>
+<tr>
+<td>
+<code>disableHealthTimeout</code>
+</td>
+<td>
+<em>
+*bool
+</em>
+</td>
+<td>
+<em>(Optional)</em>
+<p>DisableHealthTimeout if set to true, health timeout will be ignored. Leading to machine never being declared failed.</p>
+</td>
+</tr>
+<tr>
+<td>
 <code>maxEvictRetries</code>
 </td>
 <td>
diff --git a/docs/proposals/_index.md b/docs/proposals/_index.md
new file mode 100644
index 000000000..970c379bb
--- /dev/null
+++ b/docs/proposals/_index.md
@@ -0,0 +1,5 @@
+---
+title: Proposals
+weight: 3
+persona: Developers
+---
\ No newline at end of file
diff --git a/docs/todo/_index.md b/docs/todo/_index.md
new file mode 100644
index 000000000..930e852f8
--- /dev/null
+++ b/docs/todo/_index.md
@@ -0,0 +1,5 @@
+---
+title: To-Do
+weight: 4
+persona: Developers
+---
\ No newline at end of file
diff --git a/go.mod b/go.mod
index a6969079b..e9a5ce81e 100644
--- a/go.mod
+++ b/go.mod
@@ -7,8 +7,8 @@ require (
 	github.com/cenkalti/backoff/v4 v4.3.0
 	github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc
 	github.com/google/uuid v1.6.0
-	github.com/onsi/ginkgo/v2 v2.19.0
-	github.com/onsi/gomega v1.33.1
+	github.com/onsi/ginkgo/v2 v2.23.0
+	github.com/onsi/gomega v1.36.2
 	github.com/prometheus/client_golang v1.19.1
 	github.com/spf13/pflag v1.0.5
 	golang.org/x/lint v0.0.0-20210508222113-6edffad5e616
@@ -43,7 +43,7 @@ require (
 	github.com/google/gnostic-models v0.6.8 // indirect
 	github.com/google/go-cmp v0.6.0 // indirect
 	github.com/google/gofuzz v1.2.0 // indirect
-	github.com/google/pprof v0.0.0-20240525223248-4bfdf5a9a2af // indirect
+	github.com/google/pprof v0.0.0-20241210010833-40e02aabc2ad // indirect
 	github.com/imdario/mergo v0.3.12 // indirect
 	github.com/inconshreveable/mousetrap v1.1.0 // indirect
 	github.com/josharian/intern v1.0.0 // indirect
@@ -58,16 +58,16 @@ require (
 	github.com/prometheus/procfs v0.15.1 // indirect
 	github.com/spf13/cobra v1.8.1 // indirect
 	github.com/x448/float16 v0.8.4 // indirect
-	golang.org/x/mod v0.17.0 // indirect
-	golang.org/x/net v0.26.0 // indirect
+	golang.org/x/mod v0.23.0 // indirect
+	golang.org/x/net v0.35.0 // indirect
 	golang.org/x/oauth2 v0.21.0 // indirect
-	golang.org/x/sync v0.7.0 // indirect
-	golang.org/x/sys v0.21.0 // indirect
-	golang.org/x/term v0.21.0 // indirect
-	golang.org/x/text v0.16.0 // indirect
+	golang.org/x/sync v0.11.0 // indirect
+	golang.org/x/sys v0.30.0 // indirect
+	golang.org/x/term v0.29.0 // indirect
+	golang.org/x/text v0.22.0 // indirect
 	golang.org/x/time v0.3.0 // indirect
-	golang.org/x/tools v0.21.1-0.20240508182429-e35e4ccd0d2d // indirect
-	google.golang.org/protobuf v1.34.2 // indirect
+	golang.org/x/tools v0.30.0 // indirect
+	google.golang.org/protobuf v1.36.1 // indirect
 	gopkg.in/evanphx/json-patch.v4 v4.12.0 // indirect
 	gopkg.in/inf.v0 v0.9.1 // indirect
 	gopkg.in/yaml.v2 v2.4.0 // indirect
diff --git a/go.sum b/go.sum
index 4ca168c08..4eee5a452 100644
--- a/go.sum
+++ b/go.sum
@@ -45,8 +45,8 @@ github.com/google/go-cmp v0.6.0/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeN
 github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg=
 github.com/google/gofuzz v1.2.0 h1:xRy4A+RhZaiKjJ1bPfwQ8sedCA+YS2YcCHW6ec7JMi0=
 github.com/google/gofuzz v1.2.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg=
-github.com/google/pprof v0.0.0-20240525223248-4bfdf5a9a2af h1:kmjWCqn2qkEml422C2Rrd27c3VGxi6a/6HNq8QmHRKM=
-github.com/google/pprof v0.0.0-20240525223248-4bfdf5a9a2af/go.mod h1:K1liHPHnj73Fdn/EKuT8nrFqBihUSKXoLYU0BuatOYo=
+github.com/google/pprof v0.0.0-20241210010833-40e02aabc2ad h1:a6HEuzUHeKH6hwfN/ZoQgRgVIWFJljSWa/zetS2WTvg=
+github.com/google/pprof v0.0.0-20241210010833-40e02aabc2ad/go.mod h1:vavhavw2zAxS5dIdcRluK6cSGGPlZynqzFM8NdvU144=
 github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
 github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
 github.com/imdario/mergo v0.3.12 h1:b6R2BslTbIEToALKP7LxUvijTsNI9TAe80pLWN2g/HU=
@@ -75,10 +75,10 @@ github.com/modern-go/reflect2 v1.0.2 h1:xBagoLtFs94CBntxluKeaWgTMpvLxC4ur3nMaC9G
 github.com/modern-go/reflect2 v1.0.2/go.mod h1:yWuevngMOJpCy52FWWMvUC8ws7m/LJsjYzDa0/r8luk=
 github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 h1:C3w9PqII01/Oq1c1nUAm88MOHcQC9l5mIlSMApZMrHA=
 github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ=
-github.com/onsi/ginkgo/v2 v2.19.0 h1:9Cnnf7UHo57Hy3k6/m5k3dRfGTMXGvxhHFvkDTCTpvA=
-github.com/onsi/ginkgo/v2 v2.19.0/go.mod h1:rlwLi9PilAFJ8jCg9UE1QP6VBpd6/xj3SRC0d6TU0To=
-github.com/onsi/gomega v1.33.1 h1:dsYjIxxSR755MDmKVsaFQTE22ChNBcuuTWgkUDSubOk=
-github.com/onsi/gomega v1.33.1/go.mod h1:U4R44UsT+9eLIaYRB2a5qajjtQYn0hauxvRm16AVYg0=
+github.com/onsi/ginkgo/v2 v2.23.0 h1:FA1xjp8ieYDzlgS5ABTpdUDB7wtngggONc8a7ku2NqQ=
+github.com/onsi/ginkgo/v2 v2.23.0/go.mod h1:zXTP6xIp3U8aVuXN8ENK9IXRaTjFnpVB9mGmaSRvxnM=
+github.com/onsi/gomega v1.36.2 h1:koNYke6TVk6ZmnyHrCXba/T/MoLBXFjeC1PtvYgw0A8=
+github.com/onsi/gomega v1.36.2/go.mod h1:DdwyADRjrc825LhMEkD76cHR5+pUnjhUN8GlHlRPHzY=
 github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4=
 github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
 github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
@@ -126,32 +126,32 @@ golang.org/x/lint v0.0.0-20210508222113-6edffad5e616/go.mod h1:3xt1FjdF8hUf6vQPI
 golang.org/x/mod v0.1.1-0.20191105210325-c90efee705ee/go.mod h1:QqPTAvyqsEbceGzBzNggFXnrqF1CaUcvgkdR5Ot7KZg=
 golang.org/x/mod v0.2.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
 golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
-golang.org/x/mod v0.17.0 h1:zY54UmvipHiNd+pm+m0x9KhZ9hl1/7QNMyxXbc6ICqA=
-golang.org/x/mod v0.17.0/go.mod h1:hTbmBsO62+eylJbnUtE2MGJUyE7QWk4xUqPFrRgJ+7c=
+golang.org/x/mod v0.23.0 h1:Zb7khfcRGKk+kqfxFaP5tZqCnDZMjC5VtUBs87Hr6QM=
+golang.org/x/mod v0.23.0/go.mod h1:6SkKJ3Xj0I0BrPOZoBy3bdMptDDU9oJrpohJ3eWZ1fY=
 golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
 golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
 golang.org/x/net v0.0.0-20200226121028-0de0cce0169b/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
 golang.org/x/net v0.0.0-20201021035429-f5854403a974/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU=
-golang.org/x/net v0.26.0 h1:soB7SVo0PWrY4vPW/+ay0jKDNScG2X9wFeYlXIvJsOQ=
-golang.org/x/net v0.26.0/go.mod h1:5YKkiSynbBIh3p6iOc/vibscux0x38BZDkn8sCUPxHE=
+golang.org/x/net v0.35.0 h1:T5GQRQb2y08kTAByq9L4/bz8cipCdA8FbRTXewonqY8=
+golang.org/x/net v0.35.0/go.mod h1:EglIi67kWsHKlRzzVMUD93VMSWGFOMSZgxFjparz1Qk=
 golang.org/x/oauth2 v0.21.0 h1:tsimM75w1tF/uws5rbeHzIWxEqElMehnc+iW793zsZs=
 golang.org/x/oauth2 v0.21.0/go.mod h1:XYTD2NtWslqkgxebSiOHnXEap4TF09sJSc7H1sXbhtI=
 golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
 golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
 golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
-golang.org/x/sync v0.7.0 h1:YsImfSBoP9QPYL0xyKJPq0gcaJdG3rInoqxTWbfQu9M=
-golang.org/x/sync v0.7.0/go.mod h1:Czt+wKu1gCyEFDUtn0jG5QVvpJ6rzVqr5aXyt9drQfk=
+golang.org/x/sync v0.11.0 h1:GGz8+XQP4FvTTrjZPzNKTMFtSXH80RAzG+5ghFPgK9w=
+golang.org/x/sync v0.11.0/go.mod h1:Czt+wKu1gCyEFDUtn0jG5QVvpJ6rzVqr5aXyt9drQfk=
 golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
 golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20200930185726-fdedc70b468f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
-golang.org/x/sys v0.21.0 h1:rF+pYz3DAGSQAxAu1CbC7catZg4ebC4UIeIhKxBZvws=
-golang.org/x/sys v0.21.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
-golang.org/x/term v0.21.0 h1:WVXCp+/EBEHOj53Rvu+7KiT/iElMrO8ACK16SMZ3jaA=
-golang.org/x/term v0.21.0/go.mod h1:ooXLefLobQVslOqselCNF4SxFAaoS6KujMbsGzSDmX0=
+golang.org/x/sys v0.30.0 h1:QjkSwP/36a20jFYWkSue1YwXzLmsV5Gfq7Eiy72C1uc=
+golang.org/x/sys v0.30.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
+golang.org/x/term v0.29.0 h1:L6pJp37ocefwRRtYPKSWOWzOtWSxVajvz2ldH/xi3iU=
+golang.org/x/term v0.29.0/go.mod h1:6bl4lRlvVuDgSf3179VpIxBF0o10JUpXWOnI7nErv7s=
 golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
 golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
-golang.org/x/text v0.16.0 h1:a94ExnEXNtEwYLGJSIUxnWoxoRz/ZcCsV63ROupILh4=
-golang.org/x/text v0.16.0/go.mod h1:GhwF1Be+LQoKShO3cGOHzqOgRrGaYc9AvblQOmPVHnI=
+golang.org/x/text v0.22.0 h1:bofq7m3/HAFvbF51jz3Q9wLg3jkvSPuiZu/pD1XwgtM=
+golang.org/x/text v0.22.0/go.mod h1:YRoo4H8PVmsu+E3Ou7cqLVH8oXWIHVoX0jqUWALQhfY=
 golang.org/x/time v0.3.0 h1:rg5rLMjNzMS1RkNLzCG38eapWhnYLFYXDXj2gOlr8j4=
 golang.org/x/time v0.3.0/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
 golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
@@ -159,14 +159,14 @@ golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtn
 golang.org/x/tools v0.0.0-20200130002326-2f3ba24bd6e7/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28=
 golang.org/x/tools v0.0.0-20200619180055-7c47624df98f/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE=
 golang.org/x/tools v0.0.0-20210106214847-113979e3529a/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA=
-golang.org/x/tools v0.21.1-0.20240508182429-e35e4ccd0d2d h1:vU5i/LfpvrRCpgM/VPfJLg5KjxD3E+hfT1SH+d9zLwg=
-golang.org/x/tools v0.21.1-0.20240508182429-e35e4ccd0d2d/go.mod h1:aiJjzUbINMkxbQROHiO6hDPo2LHcIPhhQsa9DLh0yGk=
+golang.org/x/tools v0.30.0 h1:BgcpHewrV5AUp2G9MebG4XPFI1E2W41zU1SaqVA9vJY=
+golang.org/x/tools v0.30.0/go.mod h1:c347cR/OJfw5TI+GfX7RUPNMdDRRbjvYTS0jPyvsVtY=
 golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
 golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
 golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
 golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
-google.golang.org/protobuf v1.34.2 h1:6xV6lTsCfpGD21XK49h7MhtcApnLqkfYgPcdHftf6hg=
-google.golang.org/protobuf v1.34.2/go.mod h1:qYOHts0dSfpeUzUFpOMr/WGzszTmLH+DiWniOlNbLDw=
+google.golang.org/protobuf v1.36.1 h1:yBPeRvTftaleIgM3PZ/WBIZ7XM/eEYAaEyCwvyjq/gk=
+google.golang.org/protobuf v1.36.1/go.mod h1:9fA7Ob0pmnwhb644+1+CVWFRbNajQ6iRojtC/QF5bRE=
 gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
 gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk=
 gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q=
diff --git a/kubernetes/crds/machine.sapcloud.io_machinedeployments.yaml b/kubernetes/crds/machine.sapcloud.io_machinedeployments.yaml
index d4666f2f0..e1e155902 100644
--- a/kubernetes/crds/machine.sapcloud.io_machinedeployments.yaml
+++ b/kubernetes/crds/machine.sapcloud.io_machinedeployments.yaml
@@ -279,6 +279,11 @@ spec:
                         description: MachineCreationTimeout is the timeout after which
                           machinie creation is declared failed.
                         type: string
+                      disableHealthTimeout:
+                        description: DisableHealthTimeout if set to true, health timeout
+                          will be ignored. Leading to machine never being declared
+                          failed.
+                        type: boolean
                       drainTimeout:
                         description: MachineDraintimeout is the timeout after which
                           machine is forcefully deleted.
@@ -287,6 +292,10 @@ spec:
                         description: MachineHealthTimeout is the timeout after which
                           machine is declared unhealhty/failed.
                         type: string
+                      inPlaceUpdateTimeout:
+                        description: MachineInPlaceUpdateTimeout is the timeout after
+                          which in-place update is declared failed.
+                        type: string
                       maxEvictRetries:
                         description: MaxEvictRetries is the number of retries that
                           will be attempted while draining the node.
diff --git a/kubernetes/crds/machine.sapcloud.io_machines.yaml b/kubernetes/crds/machine.sapcloud.io_machines.yaml
index b92939c50..1e8c54bf1 100644
--- a/kubernetes/crds/machine.sapcloud.io_machines.yaml
+++ b/kubernetes/crds/machine.sapcloud.io_machines.yaml
@@ -78,6 +78,10 @@ spec:
                 description: MachineCreationTimeout is the timeout after which machinie
                   creation is declared failed.
                 type: string
+              disableHealthTimeout:
+                description: DisableHealthTimeout if set to true, health timeout will
+                  be ignored. Leading to machine never being declared failed.
+                type: boolean
               drainTimeout:
                 description: MachineDraintimeout is the timeout after which machine
                   is forcefully deleted.
@@ -86,6 +90,10 @@ spec:
                 description: MachineHealthTimeout is the timeout after which machine
                   is declared unhealhty/failed.
                 type: string
+              inPlaceUpdateTimeout:
+                description: MachineInPlaceUpdateTimeout is the timeout after which
+                  in-place update is declared failed.
+                type: string
               maxEvictRetries:
                 description: MaxEvictRetries is the number of retries that will be
                   attempted while draining the node.
diff --git a/kubernetes/crds/machine.sapcloud.io_machinesets.yaml b/kubernetes/crds/machine.sapcloud.io_machinesets.yaml
index 1b94d1664..bb57adf60 100644
--- a/kubernetes/crds/machine.sapcloud.io_machinesets.yaml
+++ b/kubernetes/crds/machine.sapcloud.io_machinesets.yaml
@@ -161,6 +161,11 @@ spec:
                         description: MachineCreationTimeout is the timeout after which
                           machinie creation is declared failed.
                         type: string
+                      disableHealthTimeout:
+                        description: DisableHealthTimeout if set to true, health timeout
+                          will be ignored. Leading to machine never being declared
+                          failed.
+                        type: boolean
                       drainTimeout:
                         description: MachineDraintimeout is the timeout after which
                           machine is forcefully deleted.
@@ -169,6 +174,10 @@ spec:
                         description: MachineHealthTimeout is the timeout after which
                           machine is declared unhealhty/failed.
                         type: string
+                      inPlaceUpdateTimeout:
+                        description: MachineInPlaceUpdateTimeout is the timeout after
+                          which in-place update is declared failed.
+                        type: string
                       maxEvictRetries:
                         description: MaxEvictRetries is the number of retries that
                           will be attempted while draining the node.
diff --git a/pkg/apis/machine/types.go b/pkg/apis/machine/types.go
index 6f7ed287c..ea6cfc07b 100644
--- a/pkg/apis/machine/types.go
+++ b/pkg/apis/machine/types.go
@@ -94,6 +94,13 @@ type MachineConfiguration struct {
 	// MachineCreationTimeout is the timeout after which machinie creation is declared failed.
 	MachineCreationTimeout *metav1.Duration
 
+	// MachineInPlaceUpdateTimeout is the timeout after which in-place update is declared failed.
+	MachineInPlaceUpdateTimeout *metav1.Duration
+
+	// DisableHealthTimeout if set to true, health timeout will be ignored. Leading to machine never being declared failed.
+	// This is intended to be used only for in-place updates.
+	DisableHealthTimeout *bool
+
 	// MaxEvictRetries is the number of retries that will be attempted while draining the node.
 	MaxEvictRetries *int32
 
diff --git a/pkg/apis/machine/v1alpha1/constants.go b/pkg/apis/machine/v1alpha1/constants.go
new file mode 100644
index 000000000..3ae59312a
--- /dev/null
+++ b/pkg/apis/machine/v1alpha1/constants.go
@@ -0,0 +1,22 @@
+// SPDX-FileCopyrightText: 2025 SAP SE or an SAP affiliate company and Gardener contributors
+//
+// SPDX-License-Identifier: Apache-2.0
+
+package v1alpha1
+
+const (
+	// AnnotationKeyMachineUpdateFailedReason is the annotation key that indicates the reason for a machine update failure.
+	AnnotationKeyMachineUpdateFailedReason = "node.machine.sapcloud.io/update-failed-reason"
+
+	// LabelKeyNodeCandidateForUpdate is the label key that indicates a node is a candidate for update.
+	LabelKeyNodeCandidateForUpdate = "node.machine.sapcloud.io/candidate-for-update"
+	// LabelKeyNodeSelectedForUpdate is the label key that indicates a node has been selected for update.
+	LabelKeyNodeSelectedForUpdate = "node.machine.sapcloud.io/selected-for-update"
+	// LabelKeyNodeUpdateResult is the label key that indicates the result of the update on the node.
+	LabelKeyNodeUpdateResult = "node.machine.sapcloud.io/update-result"
+
+	// LabelValueNodeUpdateSuccessful is the label value that indicates the update on the node has succeeded.
+	LabelValueNodeUpdateSuccessful = "successful"
+	// LabelValueNodeUpdateFailed is the label value that indicates the update on the node has failed.
+	LabelValueNodeUpdateFailed = "failed"
+)
diff --git a/pkg/apis/machine/v1alpha1/register.go b/pkg/apis/machine/v1alpha1/register.go
index 596605376..bdc4e423a 100644
--- a/pkg/apis/machine/v1alpha1/register.go
+++ b/pkg/apis/machine/v1alpha1/register.go
@@ -31,13 +31,6 @@ func Resource(resource string) schema.GroupResource {
 	return SchemeGroupVersion.WithResource(resource).GroupResource()
 }
 
-// func Init() {
-// 	// We only register manually written functions here. The registration of the
-// 	// generated functions takes place in the generated files. The separation
-// 	// makes the code compile even when the generated files are missing.
-// 	SchemeBuilder.Register(addKnownTypes)
-// }
-
 // Adds the list of known types to api.Scheme.
 func addKnownTypes(scheme *runtime.Scheme) error {
 	scheme.AddKnownTypes(SchemeGroupVersion,
diff --git a/pkg/apis/machine/v1alpha1/shared_types.go b/pkg/apis/machine/v1alpha1/shared_types.go
index 17b7c33ae..687151218 100644
--- a/pkg/apis/machine/v1alpha1/shared_types.go
+++ b/pkg/apis/machine/v1alpha1/shared_types.go
@@ -40,6 +40,15 @@ type MachineConfiguration struct {
 	// +optional
 	MachineCreationTimeout *metav1.Duration `json:"creationTimeout,omitempty"`
 
+	// MachineInPlaceUpdateTimeout is the timeout after which in-place update is declared failed.
+	// +optional
+	MachineInPlaceUpdateTimeout *metav1.Duration `json:"inPlaceUpdateTimeout,omitempty"`
+
+	// DisableHealthTimeout if set to true, health timeout will be ignored. Leading to machine never being declared failed.
+	// This is intended to be used only for in-place updates.
+	// +optional
+	DisableHealthTimeout *bool `json:"disableHealthTimeout,omitempty"`
+
 	// MaxEvictRetries is the number of retries that will be attempted while draining the node.
 	// +optional
 	MaxEvictRetries *int32 `json:"maxEvictRetries,omitempty"`
diff --git a/pkg/apis/machine/v1alpha1/zz_generated.conversion.go b/pkg/apis/machine/v1alpha1/zz_generated.conversion.go
index b34afc0b4..385ab9aae 100644
--- a/pkg/apis/machine/v1alpha1/zz_generated.conversion.go
+++ b/pkg/apis/machine/v1alpha1/zz_generated.conversion.go
@@ -530,6 +530,8 @@ func autoConvert_v1alpha1_MachineConfiguration_To_machine_MachineConfiguration(i
 	out.MachineDrainTimeout = (*metav1.Duration)(unsafe.Pointer(in.MachineDrainTimeout))
 	out.MachineHealthTimeout = (*metav1.Duration)(unsafe.Pointer(in.MachineHealthTimeout))
 	out.MachineCreationTimeout = (*metav1.Duration)(unsafe.Pointer(in.MachineCreationTimeout))
+	out.MachineInPlaceUpdateTimeout = (*metav1.Duration)(unsafe.Pointer(in.MachineInPlaceUpdateTimeout))
+	out.DisableHealthTimeout = (*bool)(unsafe.Pointer(in.DisableHealthTimeout))
 	out.MaxEvictRetries = (*int32)(unsafe.Pointer(in.MaxEvictRetries))
 	out.NodeConditions = (*string)(unsafe.Pointer(in.NodeConditions))
 	return nil
@@ -544,6 +546,8 @@ func autoConvert_machine_MachineConfiguration_To_v1alpha1_MachineConfiguration(i
 	out.MachineDrainTimeout = (*metav1.Duration)(unsafe.Pointer(in.MachineDrainTimeout))
 	out.MachineHealthTimeout = (*metav1.Duration)(unsafe.Pointer(in.MachineHealthTimeout))
 	out.MachineCreationTimeout = (*metav1.Duration)(unsafe.Pointer(in.MachineCreationTimeout))
+	out.MachineInPlaceUpdateTimeout = (*metav1.Duration)(unsafe.Pointer(in.MachineInPlaceUpdateTimeout))
+	out.DisableHealthTimeout = (*bool)(unsafe.Pointer(in.DisableHealthTimeout))
 	out.MaxEvictRetries = (*int32)(unsafe.Pointer(in.MaxEvictRetries))
 	out.NodeConditions = (*string)(unsafe.Pointer(in.NodeConditions))
 	return nil
diff --git a/pkg/apis/machine/v1alpha1/zz_generated.deepcopy.go b/pkg/apis/machine/v1alpha1/zz_generated.deepcopy.go
index 43540dd49..b3ed2a469 100644
--- a/pkg/apis/machine/v1alpha1/zz_generated.deepcopy.go
+++ b/pkg/apis/machine/v1alpha1/zz_generated.deepcopy.go
@@ -204,6 +204,16 @@ func (in *MachineConfiguration) DeepCopyInto(out *MachineConfiguration) {
 		*out = new(metav1.Duration)
 		**out = **in
 	}
+	if in.MachineInPlaceUpdateTimeout != nil {
+		in, out := &in.MachineInPlaceUpdateTimeout, &out.MachineInPlaceUpdateTimeout
+		*out = new(metav1.Duration)
+		**out = **in
+	}
+	if in.DisableHealthTimeout != nil {
+		in, out := &in.DisableHealthTimeout, &out.DisableHealthTimeout
+		*out = new(bool)
+		**out = **in
+	}
 	if in.MaxEvictRetries != nil {
 		in, out := &in.MaxEvictRetries, &out.MaxEvictRetries
 		*out = new(int32)
diff --git a/pkg/apis/machine/zz_generated.deepcopy.go b/pkg/apis/machine/zz_generated.deepcopy.go
index 953800419..d745dec9c 100644
--- a/pkg/apis/machine/zz_generated.deepcopy.go
+++ b/pkg/apis/machine/zz_generated.deepcopy.go
@@ -204,6 +204,16 @@ func (in *MachineConfiguration) DeepCopyInto(out *MachineConfiguration) {
 		*out = new(metav1.Duration)
 		**out = **in
 	}
+	if in.MachineInPlaceUpdateTimeout != nil {
+		in, out := &in.MachineInPlaceUpdateTimeout, &out.MachineInPlaceUpdateTimeout
+		*out = new(metav1.Duration)
+		**out = **in
+	}
+	if in.DisableHealthTimeout != nil {
+		in, out := &in.DisableHealthTimeout, &out.DisableHealthTimeout
+		*out = new(bool)
+		**out = **in
+	}
 	if in.MaxEvictRetries != nil {
 		in, out := &in.MaxEvictRetries, &out.MaxEvictRetries
 		*out = new(int32)
diff --git a/pkg/controller/machine_safety.go b/pkg/controller/machine_safety.go
index ffe330001..0f17f367f 100644
--- a/pkg/controller/machine_safety.go
+++ b/pkg/controller/machine_safety.go
@@ -9,6 +9,7 @@ import (
 	"context"
 	"fmt"
 
+	corev1 "k8s.io/api/core/v1"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 	"k8s.io/apimachinery/pkg/labels"
 	intstrutil "k8s.io/apimachinery/pkg/util/intstr"
@@ -26,6 +27,10 @@ const (
 	MachineDeploymentStateSync = "MachineDeploymentStateSync"
 	// UnfreezeAnnotation indicates the controllers to unfreeze this object
 	UnfreezeAnnotation = "safety.machine.sapcloud.io/unfreeze"
+	// MachineSetFreezeEvent is recorded when a machineset is frozen
+	MachineSetFreezeEvent = "FrozeMachineSet"
+	// MachineSetUnfreezeEvent is recorded when a machineset is unfrozen
+	MachineSetUnfreezeEvent = "UnfrozeMachineSet"
 )
 
 // reconcileClusterMachineSafetyOvershooting checks all machineSet/machineDeployment
@@ -330,6 +335,7 @@ func (c *controller) freezeMachineSetAndDeployment(ctx context.Context, machineS
 		}
 	}
 
+	c.recorder.Eventf(machineSet, corev1.EventTypeNormal, MachineSetFreezeEvent, "SafetyController: Froze MachineSet %s due to replica overshooting", machineSet.Name)
 	klog.V(2).Infof("SafetyController: Froze MachineSet %q due to overshooting of replicas", machineSet.Name)
 	return nil
 }
@@ -398,6 +404,7 @@ func (c *controller) unfreezeMachineSet(ctx context.Context, machineSet *v1alpha
 		return err
 	}
 
+	c.recorder.Eventf(machineSet, corev1.EventTypeNormal, MachineSetUnfreezeEvent, "SafetyController: Unfroze MachineSet %s", machineSet.Name)
 	klog.V(2).Infof("SafetyController: Unfroze MachineSet %q", machineSet.Name)
 	return nil
 }
diff --git a/pkg/controller/machineset.go b/pkg/controller/machineset.go
index cc758a10d..4176406e5 100644
--- a/pkg/controller/machineset.go
+++ b/pkg/controller/machineset.go
@@ -339,7 +339,6 @@ func (c *controller) manageReplicas(ctx context.Context, allMachines []*v1alpha1
 	var activeMachines, staleMachines []*v1alpha1.Machine
 	for _, machine := range allMachines {
 		if IsMachineActive(machine) {
-			// klog.Info("Active machine: ", machine.Name)
 			activeMachines = append(activeMachines, machine)
 		} else if IsMachineFailed(machine) {
 			staleMachines = append(staleMachines, machine)
diff --git a/pkg/openapi/api_violations.report b/pkg/openapi/api_violations.report
index d480b2b81..6634216bb 100644
--- a/pkg/openapi/api_violations.report
+++ b/pkg/openapi/api_violations.report
@@ -5,6 +5,7 @@ API rule violation: list_type_missing,github.com/gardener/machine-controller-man
 API rule violation: names_match,github.com/gardener/machine-controller-manager/pkg/apis/machine/v1alpha1,MachineConfiguration,MachineCreationTimeout
 API rule violation: names_match,github.com/gardener/machine-controller-manager/pkg/apis/machine/v1alpha1,MachineConfiguration,MachineDrainTimeout
 API rule violation: names_match,github.com/gardener/machine-controller-manager/pkg/apis/machine/v1alpha1,MachineConfiguration,MachineHealthTimeout
+API rule violation: names_match,github.com/gardener/machine-controller-manager/pkg/apis/machine/v1alpha1,MachineConfiguration,MachineInPlaceUpdateTimeout
 API rule violation: names_match,github.com/gardener/machine-controller-manager/pkg/apis/machine/v1alpha1,MachineSetStatus,Conditions
 API rule violation: names_match,github.com/gardener/machine-controller-manager/pkg/apis/machine/v1alpha1,MachineSpec,NodeTemplateSpec
 API rule violation: names_match,k8s.io/api/core/v1,AzureDiskVolumeSource,DataDiskURI
diff --git a/pkg/openapi/openapi_generated.go b/pkg/openapi/openapi_generated.go
index 6841ad277..a7f8ba82d 100644
--- a/pkg/openapi/openapi_generated.go
+++ b/pkg/openapi/openapi_generated.go
@@ -676,6 +676,19 @@ func schema_pkg_apis_machine_v1alpha1_MachineConfiguration(ref common.ReferenceC
 							Ref:         ref("k8s.io/apimachinery/pkg/apis/meta/v1.Duration"),
 						},
 					},
+					"inPlaceUpdateTimeout": {
+						SchemaProps: spec.SchemaProps{
+							Description: "MachineInPlaceUpdateTimeout is the timeout after which in-place update is declared failed.",
+							Ref:         ref("k8s.io/apimachinery/pkg/apis/meta/v1.Duration"),
+						},
+					},
+					"disableHealthTimeout": {
+						SchemaProps: spec.SchemaProps{
+							Description: "DisableHealthTimeout if set to true, health timeout will be ignored. Leading to machine never being declared failed.",
+							Type:        []string{"boolean"},
+							Format:      "",
+						},
+					},
 					"maxEvictRetries": {
 						SchemaProps: spec.SchemaProps{
 							Description: "MaxEvictRetries is the number of retries that will be attempted while draining the node.",
@@ -1443,6 +1456,19 @@ func schema_pkg_apis_machine_v1alpha1_MachineSpec(ref common.ReferenceCallback)
 							Ref:         ref("k8s.io/apimachinery/pkg/apis/meta/v1.Duration"),
 						},
 					},
+					"inPlaceUpdateTimeout": {
+						SchemaProps: spec.SchemaProps{
+							Description: "MachineInPlaceUpdateTimeout is the timeout after which in-place update is declared failed.",
+							Ref:         ref("k8s.io/apimachinery/pkg/apis/meta/v1.Duration"),
+						},
+					},
+					"disableHealthTimeout": {
+						SchemaProps: spec.SchemaProps{
+							Description: "DisableHealthTimeout if set to true, health timeout will be ignored. Leading to machine never being declared failed.",
+							Type:        []string{"boolean"},
+							Format:      "",
+						},
+					},
 					"maxEvictRetries": {
 						SchemaProps: spec.SchemaProps{
 							Description: "MaxEvictRetries is the number of retries that will be attempted while draining the node.",
diff --git a/pkg/util/provider/app/options/options.go b/pkg/util/provider/app/options/options.go
index 7aeffd09d..78778ac5b 100644
--- a/pkg/util/provider/app/options/options.go
+++ b/pkg/util/provider/app/options/options.go
@@ -67,6 +67,7 @@ func NewMCServer() *MCServer {
 				MachineCreationTimeout:                   metav1.Duration{Duration: 20 * time.Minute},
 				MachineHealthTimeout:                     metav1.Duration{Duration: 10 * time.Minute},
 				MachineDrainTimeout:                      metav1.Duration{Duration: drain.DefaultMachineDrainTimeout},
+				MachineInPlaceUpdateTimeout:              metav1.Duration{Duration: 20 * time.Minute},
 				MaxEvictRetries:                          drain.DefaultMaxEvictRetries,
 				PvDetachTimeout:                          metav1.Duration{Duration: 2 * time.Minute},
 				PvReattachTimeout:                        metav1.Duration{Duration: 90 * time.Second},
@@ -100,6 +101,7 @@ func (s *MCServer) AddFlags(fs *pflag.FlagSet) {
 	fs.DurationVar(&s.SafetyOptions.MachineCreationTimeout.Duration, "machine-creation-timeout", s.SafetyOptions.MachineCreationTimeout.Duration, "Timeout (in durartion) used while joining (during creation) of machine before it is declared as failed.")
 	fs.DurationVar(&s.SafetyOptions.MachineHealthTimeout.Duration, "machine-health-timeout", s.SafetyOptions.MachineHealthTimeout.Duration, "Timeout (in durartion) used while re-joining (in case of temporary health issues) of machine before it is declared as failed.")
 	fs.DurationVar(&s.SafetyOptions.MachineDrainTimeout.Duration, "machine-drain-timeout", drain.DefaultMachineDrainTimeout, "Timeout (in durartion) used while draining of machine before deletion, beyond which MCM forcefully deletes machine.")
+	fs.DurationVar(&s.SafetyOptions.MachineInPlaceUpdateTimeout.Duration, "machine-inplace-update-timeout", s.SafetyOptions.MachineInPlaceUpdateTimeout.Duration, "Timeout (in duration) used while updating a machine in-place, beyond which it is declared as failed.")
 	fs.Int32Var(&s.SafetyOptions.MaxEvictRetries, "machine-max-evict-retries", drain.DefaultMaxEvictRetries, "Maximum number of times evicts would be attempted on a pod before it is forcibly deleted during draining of a machine.")
 	fs.DurationVar(&s.SafetyOptions.PvDetachTimeout.Duration, "machine-pv-detach-timeout", s.SafetyOptions.PvDetachTimeout.Duration, "Timeout (in duration) used while waiting for detach of PV while evicting/deleting pods")
 	fs.DurationVar(&s.SafetyOptions.PvReattachTimeout.Duration, "machine-pv-reattach-timeout", s.SafetyOptions.PvReattachTimeout.Duration, "Timeout (in duration) used while waiting for reattach of PV onto a different node")
diff --git a/pkg/util/provider/drain/drain.go b/pkg/util/provider/drain/drain.go
index dcf660851..c8085c593 100644
--- a/pkg/util/provider/drain/drain.go
+++ b/pkg/util/provider/drain/drain.go
@@ -27,8 +27,6 @@ import (
 	"errors"
 	"fmt"
 	"io"
-	"k8s.io/apimachinery/pkg/labels"
-	"k8s.io/client-go/tools/cache"
 	"regexp"
 	"sort"
 	"strings"
@@ -41,11 +39,13 @@ import (
 	storagev1 "k8s.io/api/storage/v1"
 	apierrors "k8s.io/apimachinery/pkg/api/errors"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"k8s.io/apimachinery/pkg/labels"
 	utilerrors "k8s.io/apimachinery/pkg/util/errors"
 	"k8s.io/apimachinery/pkg/util/wait"
 	"k8s.io/client-go/kubernetes"
 	corelisters "k8s.io/client-go/listers/core/v1"
 	policyv1listers "k8s.io/client-go/listers/policy/v1"
+	"k8s.io/client-go/tools/cache"
 	"k8s.io/klog/v2"
 
 	"github.com/gardener/machine-controller-manager/pkg/util/provider/driver"
diff --git a/pkg/util/provider/machinecontroller/controller.go b/pkg/util/provider/machinecontroller/controller.go
index 7eb26d22f..baeabd5d3 100644
--- a/pkg/util/provider/machinecontroller/controller.go
+++ b/pkg/util/provider/machinecontroller/controller.go
@@ -92,6 +92,7 @@ func NewController(
 		nodeQueue:                     workqueue.NewNamedRateLimitingQueue(workqueue.DefaultControllerRateLimiter(), "node"),
 		machineClassQueue:             workqueue.NewNamedRateLimitingQueue(workqueue.DefaultControllerRateLimiter(), "machineclass"),
 		machineQueue:                  workqueue.NewNamedRateLimitingQueue(workqueue.DefaultControllerRateLimiter(), "machine"),
+		machineTerminationQueue:       workqueue.NewNamedRateLimitingQueue(workqueue.DefaultControllerRateLimiter(), "machinetermination"),
 		machineSafetyOrphanVMsQueue:   workqueue.NewNamedRateLimitingQueue(workqueue.DefaultControllerRateLimiter(), "machinesafetyorphanvms"),
 		machineSafetyAPIServerQueue:   workqueue.NewNamedRateLimitingQueue(workqueue.DefaultControllerRateLimiter(), "machinesafetyapiserver"),
 		safetyOptions:                 safetyOptions,
@@ -253,6 +254,7 @@ type controller struct {
 	nodeQueue                   workqueue.RateLimitingInterface
 	machineClassQueue           workqueue.RateLimitingInterface
 	machineQueue                workqueue.RateLimitingInterface
+	machineTerminationQueue     workqueue.RateLimitingInterface
 	machineSafetyOrphanVMsQueue workqueue.RateLimitingInterface
 	machineSafetyAPIServerQueue workqueue.RateLimitingInterface
 	// syncs
@@ -279,6 +281,7 @@ func (c *controller) Run(workers int, stopCh <-chan struct{}) {
 	defer c.secretQueue.ShutDown()
 	defer c.machineClassQueue.ShutDown()
 	defer c.machineQueue.ShutDown()
+	defer c.machineTerminationQueue.ShutDown()
 	defer c.machineSafetyOrphanVMsQueue.ShutDown()
 	defer c.machineSafetyAPIServerQueue.ShutDown()
 
@@ -307,6 +310,7 @@ func (c *controller) Run(workers int, stopCh <-chan struct{}) {
 		worker.Run(c.machineClassQueue, "ClusterMachineClass", worker.DefaultMaxRetries, true, c.reconcileClusterMachineClassKey, stopCh, &waitGroup)
 		worker.Run(c.nodeQueue, "ClusterNode", worker.DefaultMaxRetries, true, c.reconcileClusterNodeKey, stopCh, &waitGroup)
 		worker.Run(c.machineQueue, "ClusterMachine", worker.DefaultMaxRetries, true, c.reconcileClusterMachineKey, stopCh, &waitGroup)
+		worker.Run(c.machineTerminationQueue, "ClusterMachineTermination", worker.DefaultMaxRetries, true, c.reconcileClusterMachineTermination, stopCh, &waitGroup)
 		worker.Run(c.machineSafetyOrphanVMsQueue, "ClusterMachineSafetyOrphanVMs", worker.DefaultMaxRetries, true, c.reconcileClusterMachineSafetyOrphanVMs, stopCh, &waitGroup)
 		worker.Run(c.machineSafetyAPIServerQueue, "ClusterMachineAPIServer", worker.DefaultMaxRetries, true, c.reconcileClusterMachineSafetyAPIServer, stopCh, &waitGroup)
 	}
diff --git a/pkg/util/provider/machinecontroller/machine.go b/pkg/util/provider/machinecontroller/machine.go
index 0a58b6d75..d2c73a877 100644
--- a/pkg/util/provider/machinecontroller/machine.go
+++ b/pkg/util/provider/machinecontroller/machine.go
@@ -17,6 +17,7 @@ import (
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 	"k8s.io/apimachinery/pkg/labels"
 	"k8s.io/apimachinery/pkg/selection"
+	utilruntime "k8s.io/apimachinery/pkg/util/runtime"
 	"k8s.io/apimachinery/pkg/util/sets"
 	"k8s.io/client-go/tools/cache"
 	"k8s.io/klog/v2"
@@ -35,7 +36,19 @@ SECTION
 Machine controller - Machine add, update, delete watches
 */
 func (c *controller) addMachine(obj interface{}) {
-	c.enqueueMachine(obj, "handling machine obj ADD event")
+	machine, ok := obj.(*v1alpha1.Machine)
+	if !ok {
+		klog.Errorf("couldn't convert to machine resource from object")
+		return
+	}
+	// On restart of the controller process, a machine that was marked for
+	// deletion would be processed as part of an `add` event. This check
+	// ensures that its enqueued in the correct queue.
+	if machine.DeletionTimestamp != nil {
+		c.enqueueMachineTermination(machine, "handling terminating machine object ADD event")
+	} else {
+		c.enqueueMachine(obj, "handling machine obj ADD event")
+	}
 }
 
 func (c *controller) updateMachine(oldObj, newObj interface{}) {
@@ -52,11 +65,27 @@ func (c *controller) updateMachine(oldObj, newObj interface{}) {
 		return
 	}
 
-	c.enqueueMachine(newObj, "handling machine object UPDATE event")
+	if newMachine.DeletionTimestamp != nil {
+		c.enqueueMachineTermination(newMachine, "handling terminating machine object UPDATE event")
+	} else {
+		c.enqueueMachine(newObj, "handling machine object UPDATE event")
+	}
 }
 
 func (c *controller) deleteMachine(obj interface{}) {
-	c.enqueueMachine(obj, "handling machine object DELETE event")
+	machine, ok := obj.(*v1alpha1.Machine)
+	if !ok {
+		tombstone, ok := obj.(cache.DeletedFinalStateUnknown)
+		if !ok {
+			return
+		}
+		machine, ok = tombstone.Obj.(*v1alpha1.Machine)
+		if !ok {
+			utilruntime.HandleError(fmt.Errorf("Tombstone contained object that is not a Machine Deployment %#v", obj))
+			return
+		}
+	}
+	c.enqueueMachineTermination(machine, "handling terminating machine object DELETE event")
 }
 
 // getKeyForObj returns key for object, else returns false
@@ -83,6 +112,27 @@ func (c *controller) enqueueMachineAfter(obj interface{}, after time.Duration, r
 	}
 }
 
+func (c *controller) enqueueMachineTermination(machine *v1alpha1.Machine, reason string) {
+
+	if key, err := cache.DeletionHandlingMetaNamespaceKeyFunc(machine); err != nil {
+		utilruntime.HandleError(fmt.Errorf("Couldn't get key for object %#v: %v", machine, err))
+		return
+	} else {
+		klog.V(3).Infof("Adding machine object to termination queue %q, reason: %s", key, reason)
+		c.machineTerminationQueue.Add(key)
+	}
+}
+
+func (c *controller) enqueueMachineTerminationAfter(machine *v1alpha1.Machine, after time.Duration, reason string) {
+	if key, err := cache.DeletionHandlingMetaNamespaceKeyFunc(machine); err != nil {
+		utilruntime.HandleError(fmt.Errorf("Couldn't get key for object %#v: %v", machine, err))
+		return
+	} else {
+		klog.V(3).Infof("Adding machine object to termination queue %q after %s, reason: %s", key, after, reason)
+		c.machineTerminationQueue.AddAfter(key, after)
+	}
+}
+
 func (c *controller) reconcileClusterMachineKey(key string) error {
 	ctx := context.Background()
 
@@ -101,12 +151,18 @@ func (c *controller) reconcileClusterMachineKey(key string) error {
 		return err
 	}
 
+	if machine.DeletionTimestamp != nil {
+		klog.Errorf("Machine %q should be in machine termination queue", machine.Name)
+		return nil
+	}
+
 	retryPeriod, err := c.reconcileClusterMachine(ctx, machine)
 
 	var reEnqueReason = "periodic reconcile"
 	if err != nil {
 		reEnqueReason = err.Error()
 	}
+
 	c.enqueueMachineAfter(machine, time.Duration(retryPeriod), reEnqueReason)
 
 	return nil
@@ -144,18 +200,6 @@ func (c *controller) reconcileClusterMachine(ctx context.Context, machine *v1alp
 		return retry, err
 	}
 
-	if machine.DeletionTimestamp != nil {
-		// Process a delete event
-		return c.triggerDeletionFlow(
-			ctx,
-			&driver.DeleteMachineRequest{
-				Machine:      machine,
-				MachineClass: machineClass,
-				Secret:       &corev1.Secret{Data: secretData},
-			},
-		)
-	}
-
 	// Add finalizers if not present on machine object
 	retry, err = c.addMachineFinalizers(ctx, machine)
 	if err != nil {
@@ -193,6 +237,58 @@ func (c *controller) reconcileClusterMachine(ctx context.Context, machine *v1alp
 	return machineutils.LongRetry, nil
 }
 
+func (c *controller) reconcileClusterMachineTermination(key string) error {
+	ctx := context.Background()
+
+	_, name, err := cache.SplitMetaNamespaceKey(key)
+	if err != nil {
+		return err
+	}
+
+	machine, err := c.machineLister.Machines(c.namespace).Get(name)
+	if apierrors.IsNotFound(err) {
+		klog.V(4).Infof("Machine %q: Not doing work because it is not found", key)
+		return nil
+	}
+	if err != nil {
+		klog.Errorf("Machine %q: Unable to retrieve object from store: %v", key, err)
+		return err
+	}
+
+	klog.V(2).Infof("reconcileClusterMachineTermination: Start for %q with phase:%q, description:%q",
+		machine.Name, machine.Status.CurrentStatus.Phase, machine.Status.LastOperation.Description)
+	defer klog.V(2).Infof("reconcileClusterMachineTermination: Stop for %q", machine.Name)
+
+	machineClass, secretData, retry, err := c.ValidateMachineClass(ctx, &machine.Spec.Class)
+	if err != nil {
+		klog.Errorf("cannot reconcile machine %q: %s", machine.Name, err)
+		c.enqueueMachineTerminationAfter(machine, time.Duration(retry), err.Error())
+		return err
+	}
+
+	// Process a delete event
+	retryPeriod, err := c.triggerDeletionFlow(
+		ctx,
+		&driver.DeleteMachineRequest{
+			Machine:      machine,
+			MachineClass: machineClass,
+			Secret:       &corev1.Secret{Data: secretData},
+		},
+	)
+
+	if err != nil {
+		c.enqueueMachineTerminationAfter(machine, time.Duration(retryPeriod), err.Error())
+		return err
+	} else {
+		// If the informer loses connection to the API server it may need to resync.
+		// If a resource is deleted while the watch is down, the informer won’t get
+		// delete event because the object is already gone. To avoid this edge-case,
+		// a requeue is scheduled post machine deletion as well.
+		c.enqueueMachineTerminationAfter(machine, time.Duration(retryPeriod), "post-deletion reconcile")
+		return nil
+	}
+}
+
 /*
 SECTION
 Machine controller - nodeToMachine
@@ -465,30 +561,6 @@ func (c *controller) triggerCreationFlow(ctx context.Context, createMachineReque
 				nodeName = machine.Labels[v1alpha1.NodeLabelKey]
 			}
 
-		case codes.Unknown, codes.DeadlineExceeded, codes.Aborted, codes.Unavailable:
-			// GetMachineStatus() returned with one of the above error codes.
-			// Retry operation.
-			updateRetryPeriod, updateErr := c.machineStatusUpdate(
-				ctx,
-				machine,
-				v1alpha1.LastOperation{
-					Description:    "Cloud provider message - " + err.Error(),
-					State:          v1alpha1.MachineStateFailed,
-					Type:           v1alpha1.MachineOperationCreate,
-					LastUpdateTime: metav1.Now(),
-				},
-				v1alpha1.CurrentStatus{
-					Phase:          c.getCreateFailurePhase(machine),
-					LastUpdateTime: metav1.Now(),
-				},
-				machine.Status.LastKnownState,
-			)
-			if updateErr != nil {
-				return updateRetryPeriod, updateErr
-			}
-
-			return machineutils.ShortRetry, err
-
 		case codes.Uninitialized:
 			uninitializedMachine = true
 			klog.Infof("VM instance associated with machine %s was created but not initialized.", machine.Name)
@@ -498,26 +570,7 @@ func (c *controller) triggerCreationFlow(ctx context.Context, createMachineReque
 			providerID = getMachineStatusResponse.ProviderID
 
 		default:
-			updateRetryPeriod, updateErr := c.machineStatusUpdate(
-				ctx,
-				machine,
-				v1alpha1.LastOperation{
-					Description:    "Cloud provider message - " + err.Error(),
-					State:          v1alpha1.MachineStateFailed,
-					Type:           v1alpha1.MachineOperationCreate,
-					LastUpdateTime: metav1.Now(),
-				},
-				v1alpha1.CurrentStatus{
-					Phase:          c.getCreateFailurePhase(machine),
-					LastUpdateTime: metav1.Now(),
-				},
-				machine.Status.LastKnownState,
-			)
-			if updateErr != nil {
-				return updateRetryPeriod, updateErr
-			}
-
-			return machineutils.MediumRetry, err
+			return c.machineCreateErrorHandler(ctx, machine, nil, err)
 		}
 	} else {
 		if machine.Labels[v1alpha1.NodeLabelKey] == "" || machine.Spec.ProviderID == "" {
diff --git a/pkg/util/provider/machinecontroller/machine_util.go b/pkg/util/provider/machinecontroller/machine_util.go
index a0c10645c..ffd3ab411 100644
--- a/pkg/util/provider/machinecontroller/machine_util.go
+++ b/pkg/util/provider/machinecontroller/machine_util.go
@@ -498,7 +498,8 @@ func SyncMachineTaints(
 	return toBeUpdated
 }
 
-// machineCreateErrorHandler TODO
+// machineCreateErrorHandler updates the machine status based on
+// CreateMachineResponse and the error during the machine creation
 func (c *controller) machineCreateErrorHandler(ctx context.Context, machine *v1alpha1.Machine, createMachineResponse *driver.CreateMachineResponse, err error) (machineutils.RetryPeriod, error) {
 	var (
 		retryRequired  = machineutils.MediumRetry
@@ -509,6 +510,7 @@ func (c *controller) machineCreateErrorHandler(ctx context.Context, machine *v1a
 		switch machineErr.Code() {
 		case codes.Unknown, codes.DeadlineExceeded, codes.Aborted, codes.Unavailable:
 			retryRequired = machineutils.ShortRetry
+			lastKnownState = machine.Status.LastKnownState
 		}
 	}
 
diff --git a/pkg/util/provider/machineutils/utils.go b/pkg/util/provider/machineutils/utils.go
index ab13e25ca..af1458bff 100644
--- a/pkg/util/provider/machineutils/utils.go
+++ b/pkg/util/provider/machineutils/utils.go
@@ -6,10 +6,11 @@
 package machineutils
 
 import (
-	"github.com/gardener/machine-controller-manager/pkg/apis/machine/v1alpha1"
 	"time"
 
 	v1 "k8s.io/api/core/v1"
+
+	"github.com/gardener/machine-controller-manager/pkg/apis/machine/v1alpha1"
 )
 
 const (
@@ -69,6 +70,9 @@ const (
 
 	// MachineLabelKey defines the labels which contains the name of the machine of a node
 	MachineLabelKey = "node.gardener.cloud/machine-name"
+
+	// LabelKeyMachineSetScaleUpDisabled is the label key that indicates scaling up of the machine set is disabled.
+	LabelKeyMachineSetScaleUpDisabled = "node.machine.sapcloud.io/scale-up-disabled"
 )
 
 // RetryPeriod is an alias for specifying the retry period
diff --git a/pkg/util/provider/options/types.go b/pkg/util/provider/options/types.go
index 4d1df4d41..d1be4c2c9 100644
--- a/pkg/util/provider/options/types.go
+++ b/pkg/util/provider/options/types.go
@@ -87,6 +87,9 @@ type SafetyOptions struct {
 	// Timeout (in duration) used while draining of machine before deletion,
 	// beyond which it forcefully deletes machine
 	MachineDrainTimeout metav1.Duration
+	// Timeout (in duration) used while in-place updating of a machine,
+	// beyond which it is declared as failed
+	MachineInPlaceUpdateTimeout metav1.Duration
 	// Maximum number of times evicts would be attempted on a pod for it is forcibly deleted
 	// during draining of a machine.
 	MaxEvictRetries int32



Test patch:
diff --git a/pkg/test/integration/common/framework.go b/pkg/test/integration/common/framework.go
index f71d0a615..ae6bcb6d3 100644
--- a/pkg/test/integration/common/framework.go
+++ b/pkg/test/integration/common/framework.go
@@ -5,26 +5,27 @@
 package common
 
 import (
+	"bufio"
+	"bytes"
 	"context"
 	"encoding/json"
 	"fmt"
-	"github.com/gardener/machine-controller-manager/pkg/controller"
-	"io"
 	"log"
 	"os"
 	"os/exec"
+	"path"
 	"path/filepath"
-	"regexp"
 	"strconv"
 	"strings"
 	"time"
 
+	"github.com/gardener/machine-controller-manager/pkg/controller"
+
 	"github.com/onsi/ginkgo/v2"
 
 	"github.com/onsi/gomega"
 	"github.com/onsi/gomega/gexec"
 	appsV1 "k8s.io/api/apps/v1"
-	coreV1 "k8s.io/api/core/v1"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 	"k8s.io/apimachinery/pkg/types"
 	"k8s.io/apimachinery/pkg/watch"
@@ -37,6 +38,8 @@ import (
 
 const (
 	dwdIgnoreScalingAnnotation = "dependency-watchdog.gardener.cloud/ignore-scaling"
+	testMachineDeploymentName  = "test-machine-deployment"
+	testMachineName            = "test-machine"
 )
 
 var (
@@ -98,6 +101,12 @@ var (
 
 	//values for gardener-node-agent-secret-name
 	gnaSecretNameLabelValue = os.Getenv("GNA_SECRET_NAME")
+
+	// Specifies whether the CRDs should be preserved during cleanup at the end
+	preserveCRDDuringCleanup = os.Getenv("PRESERVE_CRD_AT_END")
+
+	// Are the tests running for the virtual provider
+	isVirtualProvider = os.Getenv("IS_VIRTUAL_PROVIDER")
 )
 
 // ProviderSpecPatch struct holds tags for provider, which we want to patch the  machineclass with
@@ -227,226 +236,6 @@ func (c *IntegrationTestFramework) initalizeClusters() error {
 	return nil
 }
 
-func (c *IntegrationTestFramework) prepareMcmDeployment(
-
-	mcContainerImage string,
-	mcmContainerImage string,
-	byCreating bool) error {
-
-	ctx := context.Background()
-
-	if len(machineControllerManagerDeploymentName) == 0 {
-		machineControllerManagerDeploymentName = "machine-controller-manager"
-	}
-
-	if byCreating {
-		// Create clusterroles and clusterrolebindings for control and target cluster
-		// Create secret containing target kubeconfig file
-		// Create machine-deployment using the yaml file
-		controlClusterRegexp, _ := regexp.Compile("control-cluster-role")
-		targetClusterRegexp, _ := regexp.Compile("target-cluster-role")
-		log.Printf("Creating required roles and rolebinginds")
-
-		err := filepath.Walk(
-			filepath.Join(
-				mcmRepoPath,
-				"kubernetes/deployment/out-of-tree/",
-			),
-			func(
-				path string,
-				info os.FileInfo,
-				err error,
-			) error {
-
-				if err != nil {
-					return err
-				}
-
-				if !info.IsDir() {
-					if controlClusterRegexp.MatchString(info.Name()) {
-						err = c.
-							ControlCluster.
-							ApplyFiles(
-								path,
-								controlClusterNamespace,
-							)
-					} else if targetClusterRegexp.MatchString(info.Name()) {
-						err = c.
-							TargetCluster.
-							ApplyFiles(
-								path,
-								"default",
-							)
-					}
-				}
-				return err
-			})
-
-		if err != nil {
-			return err
-		}
-
-		configFile, _ := os.ReadFile(c.TargetCluster.KubeConfigFilePath)
-		if _, err := c.ControlCluster.Clientset.
-			CoreV1().
-			Secrets(controlClusterNamespace).
-			Create(ctx,
-				&coreV1.Secret{
-					ObjectMeta: metav1.ObjectMeta{
-						Name: "machine-controller-manager",
-					},
-					Data: map[string][]byte{
-						"kubeconfig": configFile,
-					},
-					Type: coreV1.SecretTypeOpaque,
-				},
-				metav1.CreateOptions{},
-			); err != nil {
-			return err
-		}
-
-		if err := c.ControlCluster.
-			ApplyFiles(
-				"../../../kubernetes/deployment.yaml",
-				controlClusterNamespace,
-			); err != nil {
-			return err
-		}
-	}
-
-	// mcmDeploymentOrigObj holds a copy of original mcm deployment
-	result, getErr := c.ControlCluster.Clientset.
-		AppsV1().
-		Deployments(controlClusterNamespace).
-		Get(ctx, machineControllerManagerDeploymentName, metav1.GetOptions{})
-
-	if getErr != nil {
-		log.Printf("failed to get latest version of Deployment: %v", getErr)
-		return getErr
-	}
-
-	mcmDeploymentOrigObj = result
-
-	// update containers spec
-	providerSpecificRegexp, _ := regexp.Compile(mcContainerPrefix)
-
-	containers := mcmDeploymentOrigObj.Spec.Template.Spec.Containers
-
-	for i := range containers {
-		if providerSpecificRegexp.MatchString(containers[i].Image) {
-			// set container image to mcContainerImageTag as the name of the container contains provider
-			if len(mcContainerImage) != 0 {
-				containers[i].Image = mcContainerImage
-				var isOptionAvailable bool
-				for option := range containers[i].Command {
-					if strings.Contains(containers[i].Command[option], "machine-drain-timeout=") {
-						isOptionAvailable = true
-						containers[i].Command[option] = "--machine-drain-timeout=5m"
-					}
-				}
-				if !isOptionAvailable {
-					containers[i].Command = append(containers[i].Command, "--machine-drain-timeout=5m")
-				}
-			}
-		} else {
-			// set container image to mcmContainerImageTag as the name of container contains provider
-			if len(mcmContainerImage) != 0 {
-				containers[i].Image = mcmContainerImage
-			}
-
-			// set machine-safety-overshooting-period to 300ms for freeze check to succeed
-			var isOptionAvailable bool
-			for option := range containers[i].Command {
-				if strings.Contains(containers[i].Command[option], "machine-safety-overshooting-period=") {
-					isOptionAvailable = true
-					containers[i].Command[option] = "--machine-safety-overshooting-period=300ms"
-				}
-			}
-			if !isOptionAvailable {
-				containers[i].Command = append(containers[i].Command, "--machine-safety-overshooting-period=300ms")
-			}
-		}
-	}
-
-	// apply updated containers spec to mcmDeploymentObj in kubernetes cluster
-	retryErr := retry.RetryOnConflict(retry.DefaultRetry, func() error {
-		// Retrieve the latest version of Deployment before attempting to update
-		// RetryOnConflict uses exponential backoff to avoid exhausting the apiserver
-
-		mcmDeployment, getErr := c.ControlCluster.Clientset.
-			AppsV1().
-			Deployments(controlClusterNamespace).
-			Get(ctx,
-				machineControllerManagerDeploymentName,
-				metav1.GetOptions{},
-			)
-
-		if getErr != nil {
-			log.Printf("failed to get latest version of Deployment: %v", getErr)
-			return getErr
-		}
-
-		mcmDeployment.Spec.Template.Spec.Containers = containers
-		_, updateErr := c.ControlCluster.Clientset.
-			AppsV1().
-			Deployments(controlClusterNamespace).
-			Update(ctx, mcmDeployment, metav1.UpdateOptions{})
-		return updateErr
-	})
-
-	ginkgo.By("Checking controllers are ready in kubernetes cluster")
-	gomega.Eventually(
-		func() error {
-			deployment, err := c.ControlCluster.Clientset.
-				AppsV1().
-				Deployments(controlClusterNamespace).
-				Get(ctx,
-					machineControllerManagerDeploymentName,
-					metav1.GetOptions{},
-				)
-			if err != nil {
-				return err
-			}
-			if deployment.Status.ReadyReplicas == 1 {
-				pods, err := c.ControlCluster.Clientset.
-					CoreV1().
-					Pods(controlClusterNamespace).
-					List(
-						ctx,
-						metav1.ListOptions{
-							LabelSelector: "role=machine-controller-manager",
-						},
-					)
-				if err != nil {
-					return err
-				}
-				podsCount := len(pods.Items)
-				readyPods := 0
-				for _, pod := range pods.Items {
-					if len(pod.Status.ContainerStatuses) < 2 {
-						return fmt.Errorf("containers(s) not ready")
-					}
-					if !pod.Status.ContainerStatuses[0].Ready {
-						return fmt.Errorf("containers(s) not ready.\n%s", pod.Status.ContainerStatuses[0].State.String())
-					}
-					if !pod.Status.ContainerStatuses[1].Ready {
-						return fmt.Errorf("container(s) not ready.\n%s", pod.Status.ContainerStatuses[1].State.String())
-					}
-					readyPods++
-				}
-				if podsCount == readyPods {
-					return nil
-				}
-			}
-			return fmt.Errorf("deployment replicas are not ready.\n%s", deployment.Status.String())
-		},
-		c.timeout,
-		c.pollingInterval).
-		Should(gomega.BeNil())
-
-	return retryErr
-}
-
 func (c *IntegrationTestFramework) scaleMcmDeployment(replicas int32) error {
 
 	ctx := context.Background()
@@ -601,6 +390,7 @@ func (c *IntegrationTestFramework) setupMachineClass() error {
 							Labels:      machineClasses.Items[0].ObjectMeta.Labels,
 							Annotations: machineClasses.Items[0].ObjectMeta.Annotations,
 						},
+						NodeTemplate:         machineClasses.Items[0].NodeTemplate, // virtual
 						ProviderSpec:         machineClasses.Items[0].ProviderSpec,
 						SecretRef:            machineClasses.Items[0].SecretRef,
 						CredentialsSecretRef: machineClasses.Items[0].CredentialsSecretRef,
@@ -765,31 +555,26 @@ func (c *IntegrationTestFramework) runControllersLocally() {
 func (c *IntegrationTestFramework) SetupBeforeSuite() {
 	ctx := context.Background()
 	log.SetOutput(ginkgo.GinkgoWriter)
-	mcContainerImage := os.Getenv("MC_CONTAINER_IMAGE")
-	mcmContainerImage := os.Getenv("MCM_CONTAINER_IMAGE")
 
 	ginkgo.By("Checking for the clusters if provided are available")
 	gomega.Expect(c.initalizeClusters()).To(gomega.BeNil())
 
+	ginkgo.By("Killing any existing processes")
+	stopMCM(ctx)
 	//setting up MCM either locally or by deploying after checking conditions
-	if isControlSeed == "true" {
-
-		if len(mcContainerImage) != 0 || len(mcmContainerImage) != 0 {
-			ginkgo.By("Updating MCM Deployemnt")
-			gomega.Expect(c.prepareMcmDeployment(mcContainerImage, mcmContainerImage, false)).To(gomega.BeNil())
-		} else {
-			checkMcmRepoAvailable()
 
-			ginkgo.By("Scaledown existing machine controllers")
-			gomega.Expect(c.scaleMcmDeployment(0)).To(gomega.BeNil())
+	checkMcmRepoAvailable()
 
-			c.runControllersLocally()
-		}
-	} else {
+	// When running the IT with virtual provider, there's no need to
+	// scale-down MC processes since none are running in the virtual cluster.
+	// Additionally, currently whenever running IT with the virtual provider,
+	// ControlCluster is assumed to be a seed since that simplifies the setup.
+	if isControlSeed == "true" && isVirtualProvider != "true" {
+		ginkgo.By("Scaledown existing machine controllers")
+		gomega.Expect(c.scaleMcmDeployment(0)).To(gomega.BeNil())
+	} else if isControlSeed != "true" {
 		//TODO : Scaledown the MCM deployment of the actual seed of the target cluster
 
-		checkMcmRepoAvailable()
-
 		//create the custom resources in the control cluster using yaml files
 		//available in kubernetes/crds directory of machine-controller-manager repo
 		//resources to be applied are machineclass, machines, machinesets and machinedeployment
@@ -797,13 +582,14 @@ func (c *IntegrationTestFramework) SetupBeforeSuite() {
 		gomega.Expect(c.ControlCluster.
 			ApplyFiles(
 				filepath.Join(mcmRepoPath, "kubernetes/crds"), controlClusterNamespace)).To(gomega.BeNil())
+	}
 
-		if len(mcContainerImage) != 0 || len(mcmContainerImage) != 0 {
-			ginkgo.By("Creating MCM Deployemnt")
-			gomega.Expect(c.prepareMcmDeployment(mcContainerImage, mcmContainerImage, true)).To(gomega.BeNil())
-		} else {
-			c.runControllersLocally()
-		}
+	c.runControllersLocally()
+
+	ginkgo.By("Cleaning any old resources")
+	if c.ControlCluster.McmClient != nil {
+		timeout := int64(900)
+		c.cleanTestResources(ctx, timeout)
 	}
 
 	ginkgo.By("Setup MachineClass")
@@ -826,30 +612,21 @@ func (c *IntegrationTestFramework) SetupBeforeSuite() {
 			machineClass.SecretRef,
 			machineClass.CredentialsSecretRef,
 		)
-	gomega.Expect(err).
-		NotTo(
-			gomega.HaveOccurred(),
-		)
+	gomega.Expect(err).NotTo(gomega.HaveOccurred())
 
-	ginkgo.By(
-		"Initializing orphan resource tracker",
-	)
+	ginkgo.By("Initializing orphan resource tracker")
 	err = c.resourcesTracker.InitializeResourcesTracker(machineClass, secretData, clusterName)
-
 	gomega.Expect(err).NotTo(gomega.HaveOccurred())
 	log.Println("orphan resource tracker initialized")
-
 }
 
 // BeforeEachCheck checks if all the nodes are ready and the controllers are runnings
 func (c *IntegrationTestFramework) BeforeEachCheck() {
 	ginkgo.BeforeEach(func() {
-		if len(os.Getenv("MC_CONTAINER_IMAGE")) == 0 && len(os.Getenv("MCM_CONTAINER_IMAGE")) == 0 {
-			ginkgo.By("Checking machineController process is running")
-			gomega.Expect(mcsession.ExitCode()).Should(gomega.Equal(-1))
-			ginkgo.By("Checking machineControllerManager process is running")
-			gomega.Expect(mcmsession.ExitCode()).Should(gomega.Equal(-1))
-		}
+		ginkgo.By("Checking machineController process is running")
+		gomega.Expect(mcsession.ExitCode()).Should(gomega.Equal(-1))
+		ginkgo.By("Checking machineControllerManager process is running")
+		gomega.Expect(mcmsession.ExitCode()).Should(gomega.Equal(-1))
 		ginkgo.By("Checking nodes in target cluster are healthy")
 		gomega.Eventually(
 			c.TargetCluster.GetNumberOfReadyNodes,
@@ -870,6 +647,12 @@ func (c *IntegrationTestFramework) ControllerTests() {
 		var initialNodes int16
 		ginkgo.Context("creation", func() {
 			ginkgo.It("should not lead to any errors and add 1 more node in target cluster", func() {
+				// In case of existing deployments creating nodes when starting virtual
+				// provider, the change in node count can be >1, this delay prevents
+				// checking node count immediately to allow for a correct initial count
+				if isVirtualProvider == "true" {
+					time.Sleep(2 * time.Second)
+				}
 				// Probe nodes currently available in target cluster
 				initialNodes = c.TargetCluster.GetNumberOfNodes()
 				ginkgo.By("Checking for errors")
@@ -903,7 +686,7 @@ func (c *IntegrationTestFramework) ControllerTests() {
 							c.ControlCluster.McmClient.
 								MachineV1alpha1().
 								Machines(controlClusterNamespace).
-								Delete(ctx, "test-machine", metav1.DeleteOptions{})).
+								Delete(ctx, testMachineName, metav1.DeleteOptions{})).
 							Should(gomega.BeNil(), "No Errors while deleting machine")
 
 						ginkgo.By("Waiting until test-machine machine object is deleted")
@@ -913,7 +696,7 @@ func (c *IntegrationTestFramework) ControllerTests() {
 							c.pollingInterval).
 							Should(gomega.BeTrue())
 
-						ginkgo.By("Waiting until number of ready nodes is equal to number of initial  nodes")
+						ginkgo.By("Waiting until number of ready nodes is equal to number of initial nodes")
 						gomega.Eventually(
 							c.TargetCluster.GetNumberOfNodes,
 							c.timeout,
@@ -957,6 +740,7 @@ func (c *IntegrationTestFramework) ControllerTests() {
 	// Testcase #02 | machine deployment
 	ginkgo.Describe("machine deployment resource", func() {
 		var initialNodes int16 // initialization should be part of creation test logic
+		initialEventCount := 0
 		ginkgo.Context("creation with replicas=0, scale up with replicas=1", func() {
 			ginkgo.It("should not lead to errors and add 1 more node to target cluster", func() {
 				var machineDeployment *v1alpha1.MachineDeployment
@@ -967,14 +751,14 @@ func (c *IntegrationTestFramework) ControllerTests() {
 				gomega.Expect(c.ControlCluster.CreateMachineDeployment(controlClusterNamespace, gnaSecretNameLabelValue, 0)).To(gomega.BeNil())
 
 				ginkgo.By("Waiting for Machine Set to be created")
-				gomega.Eventually(func() int { return c.getNumberOfMachineSets(ctx, controlClusterNamespace) }, c.timeout, c.pollingInterval).Should(gomega.BeNumerically("==", 1))
+				gomega.Eventually(func() int { return len(c.getTestMachineSets(ctx, controlClusterNamespace)) }, c.timeout, c.pollingInterval).Should(gomega.BeNumerically("==", 1))
 
 				ginkgo.By("Updating machineDeployment replicas to 1")
 				retryErr := retry.RetryOnConflict(retry.DefaultRetry, func() error {
 					machineDeployment, _ = c.ControlCluster.McmClient.
 						MachineV1alpha1().
 						MachineDeployments(controlClusterNamespace).
-						Get(ctx, "test-machine-deployment", metav1.GetOptions{})
+						Get(ctx, testMachineDeploymentName, metav1.GetOptions{})
 					machineDeployment.Spec.Replicas = 1
 					_, updateErr := c.ControlCluster.McmClient.
 						MachineV1alpha1().
@@ -991,7 +775,7 @@ func (c *IntegrationTestFramework) ControllerTests() {
 					machineDeployment, err = c.ControlCluster.McmClient.
 						MachineV1alpha1().
 						MachineDeployments(controlClusterNamespace).
-						Get(ctx, "test-machine-deployment", metav1.GetOptions{})
+						Get(ctx, testMachineDeploymentName, metav1.GetOptions{})
 					gomega.Expect(err).NotTo(gomega.HaveOccurred())
 					if len(machineDeployment.Status.Conditions) != 2 {
 						return false
@@ -1021,6 +805,8 @@ func (c *IntegrationTestFramework) ControllerTests() {
 					c.timeout,
 					c.pollingInterval).
 					Should(gomega.BeNumerically("==", initialNodes+1))
+				ginkgo.By("Fetching initial number of machineset freeze events")
+				initialEventCount = c.machineSetFreezeEventCount(ctx, controlClusterNamespace)
 			})
 		})
 		ginkgo.Context("scale-up with replicas=6", func() {
@@ -1029,7 +815,7 @@ func (c *IntegrationTestFramework) ControllerTests() {
 					machineDployment, _ := c.ControlCluster.McmClient.
 						MachineV1alpha1().
 						MachineDeployments(controlClusterNamespace).
-						Get(ctx, "test-machine-deployment", metav1.GetOptions{})
+						Get(ctx, testMachineDeploymentName, metav1.GetOptions{})
 					machineDployment.Spec.Replicas = 6
 					_, updateErr := c.ControlCluster.McmClient.
 						MachineV1alpha1().
@@ -1059,7 +845,7 @@ func (c *IntegrationTestFramework) ControllerTests() {
 					machineDployment, _ := c.ControlCluster.McmClient.
 						MachineV1alpha1().
 						MachineDeployments(controlClusterNamespace).
-						Get(ctx, "test-machine-deployment", metav1.GetOptions{})
+						Get(ctx, testMachineDeploymentName, metav1.GetOptions{})
 					machineDployment.Spec.Replicas = 2
 					_, updateErr := c.ControlCluster.McmClient.
 						MachineV1alpha1().
@@ -1084,68 +870,9 @@ func (c *IntegrationTestFramework) ControllerTests() {
 			})
 			// rapid scaling back to 2, should lead to freezing and unfreezing
 			ginkgo.It("should freeze and unfreeze machineset temporarily", func() {
-				if mcsession == nil {
-					// controllers running in pod
-					// Create log file from container log
-					mcmOutputFile, err := rotateOrAppendLogFile(mcmLogFile, true)
-					gomega.Expect(err).NotTo(gomega.HaveOccurred())
-					mcOutputFile, err := rotateOrAppendLogFile(mcLogFile, true)
-					gomega.Expect(err).NotTo(gomega.HaveOccurred())
-					ginkgo.By("Reading container log is leading to no errors")
-					podList, err := c.ControlCluster.Clientset.
-						CoreV1().
-						Pods(controlClusterNamespace).
-						List(
-							ctx,
-							metav1.ListOptions{
-								LabelSelector: "role=machine-controller-manager",
-							},
-						)
-					gomega.Expect(err).NotTo(gomega.HaveOccurred())
-
-					mcmPod := podList.Items[0]
-					providerSpecificRegexp, _ := regexp.Compile(mcContainerPrefix)
-					containers := mcmPod.Spec.Containers
-
-					for i := range containers {
-						if providerSpecificRegexp.Match([]byte(containers[i].Image)) {
-							readCloser, err := c.ControlCluster.Clientset.CoreV1().
-								Pods(controlClusterNamespace).
-								GetLogs(
-									mcmPod.Name,
-									&coreV1.PodLogOptions{
-										Container: containers[i].Name,
-									},
-								).Stream(ctx)
-							gomega.Expect(err).NotTo(gomega.HaveOccurred())
-							_, err = io.Copy(mcOutputFile, readCloser)
-							gomega.Expect(err).NotTo(gomega.HaveOccurred())
-						} else {
-							readCloser, err := c.ControlCluster.Clientset.CoreV1().
-								Pods(controlClusterNamespace).
-								GetLogs(mcmPod.Name, &coreV1.PodLogOptions{
-									Container: containers[i].Name,
-								}).Stream(ctx)
-							gomega.Expect(err).NotTo(gomega.HaveOccurred())
-							_, err = io.Copy(mcmOutputFile, readCloser)
-							gomega.Expect(err).NotTo(gomega.HaveOccurred())
-						}
-					}
-				}
-
-				ginkgo.By("Searching for Froze in mcm log file")
-				frozeRegexp, _ := regexp.Compile(` Froze MachineSet`)
-				gomega.Eventually(func() bool {
-					data, _ := os.ReadFile(mcmLogFile) // #nosec G304 -- Test only
-					return frozeRegexp.Match(data)
-				}, c.timeout, c.pollingInterval).Should(gomega.BeTrue())
-
-				ginkgo.By("Searching Unfroze in mcm log file")
-				unfrozeRegexp, _ := regexp.Compile(` Unfroze MachineSet`)
-				gomega.Eventually(func() bool {
-					data, _ := os.ReadFile(mcmLogFile) // #nosec G304 -- Test only
-					return unfrozeRegexp.Match(data)
-				}, c.timeout, c.pollingInterval).Should(gomega.BeTrue())
+				gomega.Eventually(func() int {
+					return c.machineSetFreezeEventCount(ctx, controlClusterNamespace)
+				}, c.timeout, c.pollingInterval).Should(gomega.Equal(initialEventCount + 2))
 			})
 		})
 		ginkgo.Context("updation to v2 machine-class and replicas=4", func() {
@@ -1156,7 +883,7 @@ func (c *IntegrationTestFramework) ControllerTests() {
 					machineDployment, _ := c.ControlCluster.McmClient.
 						MachineV1alpha1().
 						MachineDeployments(controlClusterNamespace).
-						Get(ctx, "test-machine-deployment", metav1.GetOptions{})
+						Get(ctx, testMachineDeploymentName, metav1.GetOptions{})
 					machineDployment.Spec.Template.Spec.Class.Name = testMachineClassResources[1]
 					machineDployment.Spec.Replicas = 4
 					_, updateErr := c.ControlCluster.McmClient.
@@ -1172,7 +899,7 @@ func (c *IntegrationTestFramework) ControllerTests() {
 					machineDeployment, err := c.ControlCluster.McmClient.
 						MachineV1alpha1().
 						MachineDeployments(controlClusterNamespace).
-						Get(ctx, "test-machine-deployment", metav1.GetOptions{})
+						Get(ctx, testMachineDeploymentName, metav1.GetOptions{})
 					if err != nil {
 						log.Println("Failed to get machinedeployment object")
 					}
@@ -1183,7 +910,7 @@ func (c *IntegrationTestFramework) ControllerTests() {
 					machineDeployment, err := c.ControlCluster.McmClient.
 						MachineV1alpha1().
 						MachineDeployments(controlClusterNamespace).
-						Get(ctx, "test-machine-deployment", metav1.GetOptions{})
+						Get(ctx, testMachineDeploymentName, metav1.GetOptions{})
 					if err != nil {
 						log.Println("Failed to get machinedeployment object")
 					}
@@ -1209,7 +936,7 @@ func (c *IntegrationTestFramework) ControllerTests() {
 					_, err := c.ControlCluster.McmClient.
 						MachineV1alpha1().
 						MachineDeployments(controlClusterNamespace).
-						Get(ctx, "test-machine-deployment", metav1.GetOptions{})
+						Get(ctx, testMachineDeploymentName, metav1.GetOptions{})
 					if err == nil {
 						ginkgo.By("Checking for errors")
 						gomega.Expect(
@@ -1218,7 +945,7 @@ func (c *IntegrationTestFramework) ControllerTests() {
 								MachineDeployments(controlClusterNamespace).
 								Delete(
 									ctx,
-									"test-machine-deployment",
+									testMachineDeploymentName,
 									metav1.DeleteOptions{},
 								)).
 							Should(gomega.BeNil())
@@ -1263,134 +990,62 @@ func (c *IntegrationTestFramework) Cleanup() {
 
 	ginkgo.By("Running Cleanup")
 	//running locally, means none of the image is specified
-	if len(os.Getenv("MC_CONTAINER_IMAGE")) == 0 && len(os.Getenv("MCM_CONTAINER_IMAGE")) == 0 {
-		for i := 0; i < 5; i++ {
-			if mcsession.ExitCode() != -1 {
-				ginkgo.By("Restarting Machine Controller ")
-				outputFile, err := rotateOrAppendLogFile(mcLogFile, false)
-				gomega.Expect(err).NotTo(gomega.HaveOccurred())
-				_, err = outputFile.WriteString("\n------------RESTARTED MC------------\n")
-				gomega.Expect(err).NotTo(gomega.HaveOccurred())
-				args := strings.Fields(
-					fmt.Sprintf(
-						"make --directory=%s start CONTROL_KUBECONFIG=%s TARGET_KUBECONFIG=%s CONTROL_NAMESPACE=%s LEADER_ELECT=false ",
-						"../../..",
-						c.ControlCluster.KubeConfigFilePath,
-						c.TargetCluster.KubeConfigFilePath,
-						controlClusterNamespace),
-				)
-				mcsession, err = gexec.Start(exec.Command(args[0], args[1:]...), outputFile, outputFile) // #nosec G204 -- Test only
-				gomega.Expect(err).NotTo(gomega.HaveOccurred())
-				break
-			}
-			time.Sleep(2 * time.Second)
+	for range 5 {
+		if mcsession.ExitCode() != -1 {
+			ginkgo.By("Restarting Machine Controller ")
+			outputFile, err := rotateOrAppendLogFile(mcLogFile, false)
+			gomega.Expect(err).NotTo(gomega.HaveOccurred())
+			_, err = outputFile.WriteString("\n------------RESTARTED MC------------\n")
+			gomega.Expect(err).NotTo(gomega.HaveOccurred())
+			args := strings.Fields(
+				fmt.Sprintf(
+					"make --directory=%s start CONTROL_KUBECONFIG=%s TARGET_KUBECONFIG=%s CONTROL_NAMESPACE=%s LEADER_ELECT=false ",
+					"../../..",
+					c.ControlCluster.KubeConfigFilePath,
+					c.TargetCluster.KubeConfigFilePath,
+					controlClusterNamespace),
+			)
+			mcsession, err = gexec.Start(exec.Command(args[0], args[1:]...), outputFile, outputFile) // #nosec G204 -- Test only
+			gomega.Expect(err).NotTo(gomega.HaveOccurred())
+			break
 		}
-		for i := 0; i < 5; i++ {
-			if mcmsession.ExitCode() != -1 {
-				ginkgo.By("Restarting Machine Controller Manager")
-				outputFile, err := rotateOrAppendLogFile(mcmLogFile, false)
-				gomega.Expect(err).NotTo(gomega.HaveOccurred())
-				_, err = outputFile.WriteString("\n------------RESTARTED MCM------------\n")
-				gomega.Expect(err).NotTo(gomega.HaveOccurred())
-				args := strings.Fields(
-					fmt.Sprintf(
-						"make --directory=%s start CONTROL_KUBECONFIG=%s TARGET_KUBECONFIG=%s CONTROL_NAMESPACE=%s LEADER_ELECT=false MACHINE_SAFETY_OVERSHOOTING_PERIOD=300ms",
-						mcmRepoPath,
-						c.ControlCluster.KubeConfigFilePath,
-						c.TargetCluster.KubeConfigFilePath,
-						controlClusterNamespace),
-				)
-				mcmsession, err = gexec.Start(exec.Command(args[0], args[1:]...), outputFile, outputFile) // #nosec G204 -- Test only
-				gomega.Expect(err).NotTo(gomega.HaveOccurred())
-				break
-			}
-			time.Sleep(2 * time.Second)
+		time.Sleep(2 * time.Second)
+	}
+	for range 5 {
+		if mcmsession.ExitCode() != -1 {
+			ginkgo.By("Restarting Machine Controller Manager")
+			outputFile, err := rotateOrAppendLogFile(mcmLogFile, false)
+			gomega.Expect(err).NotTo(gomega.HaveOccurred())
+			_, err = outputFile.WriteString("\n------------RESTARTED MCM------------\n")
+			gomega.Expect(err).NotTo(gomega.HaveOccurred())
+			args := strings.Fields(
+				fmt.Sprintf(
+					"make --directory=%s start CONTROL_KUBECONFIG=%s TARGET_KUBECONFIG=%s CONTROL_NAMESPACE=%s LEADER_ELECT=false MACHINE_SAFETY_OVERSHOOTING_PERIOD=300ms",
+					mcmRepoPath,
+					c.ControlCluster.KubeConfigFilePath,
+					c.TargetCluster.KubeConfigFilePath,
+					controlClusterNamespace),
+			)
+			mcmsession, err = gexec.Start(exec.Command(args[0], args[1:]...), outputFile, outputFile) // #nosec G204 -- Test only
+			gomega.Expect(err).NotTo(gomega.HaveOccurred())
+			break
 		}
+		time.Sleep(2 * time.Second)
 	}
 
-	if c.ControlCluster.McmClient != nil {
+	if preserveCRDDuringCleanup == "true" {
+		log.Printf("Preserve CRD: %s\n", preserveCRDDuringCleanup)
+	} else if c.ControlCluster.McmClient != nil {
 		timeout := int64(900)
-		// Check and delete machinedeployment resource
-		_, err := c.ControlCluster.McmClient.
-			MachineV1alpha1().
-			MachineDeployments(controlClusterNamespace).
-			Get(ctx, "test-machine-deployment", metav1.GetOptions{})
-		if err == nil {
-			log.Println("deleting test-machine-deployment")
-			watchMachinesDepl, _ := c.ControlCluster.McmClient.
-				MachineV1alpha1().
-				MachineDeployments(controlClusterNamespace).
-				Watch(ctx, metav1.ListOptions{TimeoutSeconds: &timeout}) //ResourceVersion: machineDeploymentObj.ResourceVersion
-			for event := range watchMachinesDepl.ResultChan() {
-				gomega.Expect(c.ControlCluster.McmClient.
-					MachineV1alpha1().
-					MachineDeployments(controlClusterNamespace).
-					Delete(ctx, "test-machine-deployment", metav1.DeleteOptions{})).To(gomega.Or(gomega.Succeed(), matchers.BeNotFoundError()))
-				if event.Type == watch.Deleted {
-					watchMachinesDepl.Stop()
-					log.Println("machinedeployment deleted")
-				}
-			}
-		} else {
-			log.Println(err.Error())
-		}
-		// Check and delete machine resource
-		_, err = c.ControlCluster.McmClient.
-			MachineV1alpha1().
-			Machines(controlClusterNamespace).
-			Get(ctx, "test-machine", metav1.GetOptions{})
-		if err == nil {
-			log.Println("deleting test-machine")
-			watchMachines, _ := c.ControlCluster.McmClient.
-				MachineV1alpha1().
-				Machines(controlClusterNamespace).
-				Watch(ctx, metav1.ListOptions{TimeoutSeconds: &timeout}) //ResourceVersion: machineObj.ResourceVersion
-			for event := range watchMachines.ResultChan() {
-				gomega.Expect(c.ControlCluster.McmClient.
-					MachineV1alpha1().
-					Machines(controlClusterNamespace).
-					Delete(ctx, "test-machine", metav1.DeleteOptions{})).To(gomega.Or(gomega.Succeed(), matchers.BeNotFoundError()))
-				if event.Type == watch.Deleted {
-					watchMachines.Stop()
-					log.Println("machine deleted")
-				}
-			}
-		} else {
-			log.Println(err.Error())
-		}
+		c.cleanTestResources(ctx, timeout)
+	}
 
-		for _, machineClassName := range testMachineClassResources {
-			// Check and delete machine class resource
-			_, err = c.ControlCluster.McmClient.
-				MachineV1alpha1().
-				MachineClasses(controlClusterNamespace).
-				Get(ctx, machineClassName, metav1.GetOptions{})
-			if err == nil {
-				log.Printf("deleting %s machineclass", machineClassName)
-				watchMachineClass, _ := c.ControlCluster.McmClient.
-					MachineV1alpha1().
-					MachineClasses(controlClusterNamespace).
-					Watch(ctx, metav1.ListOptions{TimeoutSeconds: &timeout})
-				for event := range watchMachineClass.ResultChan() {
-					gomega.Expect(c.ControlCluster.McmClient.
-						MachineV1alpha1().
-						MachineClasses(controlClusterNamespace).
-						Delete(ctx, machineClassName, metav1.DeleteOptions{})).To(gomega.Or(gomega.Succeed(), matchers.BeNotFoundError()))
-					if event.Type == watch.Deleted {
-						watchMachineClass.Stop()
-						log.Println("machineclass deleted")
-					}
-				}
-			} else {
-				log.Println(err.Error())
-			}
-		}
+	ginkgo.By("Killing any existing processes")
+	stopMCM(ctx)
 
-	}
 	if isControlSeed == "true" {
-		// scale back up the MCM deployment to 1 in the Control Cluster
 		// This is needed when IT suite runs locally against a Control & Target cluster
-
+		ginkgo.By("Scale back the existing machine controllers")
 		if err := c.scaleMcmDeployment(1); err != nil {
 			log.Println(err.Error())
 		}
@@ -1403,71 +1058,227 @@ func (c *IntegrationTestFramework) Cleanup() {
 		); err != nil {
 			log.Printf("Error occured while deleting crds. %s", err.Error())
 		}
+	}
 
-		if len(os.Getenv("MC_CONTAINER_IMAGE")) != 0 || len(os.Getenv("MCM_CONTAINER_IMAGE")) != 0 {
-			log.Println("Deleting Clusterroles")
+}
 
-			if err := c.ControlCluster.RbacClient.ClusterRoles().Delete(
-				ctx,
-				"machine-controller-manager-control",
-				metav1.DeleteOptions{},
-			); err != nil {
-				log.Printf("Error occured while deleting clusterrole. %s", err.Error())
-			}
+func (c *IntegrationTestFramework) cleanTestResources(ctx context.Context, timeout int64) {
+	// Check and delete machinedeployment resource
+	if err := c.cleanMachineDeployment(ctx, testMachineDeploymentName, timeout); err != nil {
+		log.Println(err.Error())
+	}
+	// Check and delete machine resource
+	if err := c.cleanMachine(ctx, testMachineName, timeout); err != nil {
+		log.Println(err.Error())
+	}
 
-			if err := c.ControlCluster.RbacClient.ClusterRoleBindings().Delete(
-				ctx,
-				"machine-controller-manager-control",
-				metav1.DeleteOptions{},
-			); err != nil {
-				log.Printf("Error occured while deleting clusterrolebinding . %s", err.Error())
-			}
+	for _, machineClassName := range testMachineClassResources {
+		// Check and delete machine class resource
+		if err := c.cleanMachineClass(ctx, machineClassName, timeout); err != nil {
+			log.Println(err.Error())
+		}
+	}
+}
 
-			if err := c.TargetCluster.RbacClient.ClusterRoles().Delete(
-				ctx,
-				"machine-controller-manager-target",
-				metav1.DeleteOptions{},
-			); err != nil {
-				log.Printf("Error occured while deleting clusterrole . %s", err.Error())
-			}
+func (c *IntegrationTestFramework) cleanMachineDeployment(ctx context.Context, name string, timeout int64) error {
+	_, err := c.ControlCluster.McmClient.
+		MachineV1alpha1().
+		MachineDeployments(controlClusterNamespace).
+		Get(ctx, name, metav1.GetOptions{})
+	if err != nil {
+		return err
+	}
+	log.Printf("deleting %s\n", name)
+	selector := fmt.Sprintf("metadata.name=%s", name)
+	watchMachinesDepl, _ := c.ControlCluster.McmClient.
+		MachineV1alpha1().
+		MachineDeployments(controlClusterNamespace).
+		Watch(ctx, metav1.ListOptions{
+			TimeoutSeconds: &timeout,
+			FieldSelector:  selector,
+		})
+	for event := range watchMachinesDepl.ResultChan() {
+		gomega.Expect(c.ControlCluster.McmClient.
+			MachineV1alpha1().
+			MachineDeployments(controlClusterNamespace).
+			Delete(ctx, name, metav1.DeleteOptions{})).To(gomega.Or(gomega.Succeed(), matchers.BeNotFoundError()))
+		if event.Type == watch.Deleted {
+			watchMachinesDepl.Stop()
+			log.Println("machinedeployment deleted")
+		}
+	}
+	return nil
+}
 
-			if err := c.TargetCluster.RbacClient.ClusterRoleBindings().Delete(
-				ctx,
-				"machine-controller-manager-target",
-				metav1.DeleteOptions{},
-			); err != nil {
-				log.Printf("Error occured while deleting clusterrolebinding . %s", err.Error())
-			}
+func (c *IntegrationTestFramework) cleanMachine(ctx context.Context, name string, timeout int64) error {
+	_, err := c.ControlCluster.McmClient.
+		MachineV1alpha1().
+		Machines(controlClusterNamespace).
+		Get(ctx, name, metav1.GetOptions{})
+	if err != nil {
+		return err
+	}
+	log.Printf("deleting %s\n", name)
+	selector := fmt.Sprintf("metadata.name=%s", name)
+	watchMachines, _ := c.ControlCluster.McmClient.
+		MachineV1alpha1().
+		Machines(controlClusterNamespace).
+		Watch(ctx, metav1.ListOptions{
+			TimeoutSeconds: &timeout,
+			FieldSelector:  selector,
+		}) //ResourceVersion: machineObj.ResourceVersion
+	for event := range watchMachines.ResultChan() {
+		gomega.Expect(c.ControlCluster.McmClient.
+			MachineV1alpha1().
+			Machines(controlClusterNamespace).
+			Delete(ctx, name, metav1.DeleteOptions{})).To(gomega.Or(gomega.Succeed(), matchers.BeNotFoundError()))
+		if event.Type == watch.Deleted {
+			watchMachines.Stop()
+			log.Println("machine deleted")
+		}
+	}
+	return nil
+}
 
-			log.Println("Deleting MCM deployment")
+func (c *IntegrationTestFramework) cleanMachineClass(ctx context.Context, machineClassName string, timeout int64) error {
+	_, err := c.ControlCluster.McmClient.
+		MachineV1alpha1().
+		MachineClasses(controlClusterNamespace).
+		Get(ctx, machineClassName, metav1.GetOptions{})
+	if err != nil {
+		return err
+	}
 
-			gomega.Expect(c.ControlCluster.Clientset.
-				CoreV1().
-				Secrets(controlClusterNamespace).
-				Delete(ctx,
-					"machine-controller-manager",
-					metav1.DeleteOptions{})).To(gomega.Or(gomega.Succeed(), matchers.BeNotFoundError()))
-			gomega.Expect(c.ControlCluster.Clientset.
-				AppsV1().
-				Deployments(controlClusterNamespace).
-				Delete(ctx,
-					machineControllerManagerDeploymentName,
-					metav1.DeleteOptions{})).To(gomega.Or(gomega.Succeed(), matchers.BeNotFoundError()))
+	log.Printf("deleting %s machineclass", machineClassName)
+	selector := fmt.Sprintf("metadata.name=%s", machineClassName)
+	watchMachineClass, _ := c.ControlCluster.McmClient.
+		MachineV1alpha1().
+		MachineClasses(controlClusterNamespace).
+		Watch(ctx, metav1.ListOptions{
+			TimeoutSeconds: &timeout,
+			FieldSelector:  selector,
+		})
+	for event := range watchMachineClass.ResultChan() {
+		gomega.Expect(c.ControlCluster.McmClient.
+			MachineV1alpha1().
+			MachineClasses(controlClusterNamespace).
+			Delete(ctx, machineClassName, metav1.DeleteOptions{})).To(gomega.Or(gomega.Succeed(), matchers.BeNotFoundError()))
+		if event.Type == watch.Deleted {
+			watchMachineClass.Stop()
+			log.Println("machineclass deleted")
+		}
+	}
+	return nil
+}
+
+func stopMCM(ctx context.Context) {
+	processesToKill := []string{
+		"machine-controller-manager --control-kubeconfig", // virtual
+		"machine-controller --control-kubeconfig",         // virtual
+		"controller_manager --control-kubeconfig",
+		"main --control-kubeconfig",
+		"go run cmd/machine-controller/main.go",
+		"go run cmd/machine-controller-manager/controller_manager.go",
+		"make --directory=../../..",
+	}
+	for _, proc := range processesToKill {
+		pids, err := findAndKillProcess(ctx, proc)
+		if err != nil {
+			log.Println(err.Error())
+		}
+		if len(pids) > 0 {
+			log.Printf("stopMCM killed MCM process(es) with pid(s): %v\n", pids)
 		}
 	}
+}
 
+func findAndKillProcess(ctx context.Context, prefix string) (pids []int, err error) {
+	pids, err = findPidsByPrefix(ctx, prefix)
+	if len(pids) != 0 {
+		var proc *os.Process
+		for _, pid := range pids {
+			proc, err = os.FindProcess(pid)
+			if err != nil {
+				err = fmt.Errorf("failed to find process with PID %d: %v", pid, err)
+				return
+			}
+			err = proc.Kill()
+			if err != nil {
+				err = fmt.Errorf("failed to kill process with PID %d: %v", pid, err)
+				return
+			}
+		}
+	}
+	return
 }
 
-func (c *IntegrationTestFramework) getNumberOfMachineSets(ctx context.Context, namespace string) int {
+func findPidsByPrefix(ctx context.Context, prefix string) (pids []int, err error) {
+	cmd := exec.CommandContext(ctx, "ps", "-e", "-o", "pid,args")
+	psOutput, err := cmd.Output()
+	if err != nil {
+		log.Printf("FindProcess could not run ps command: %v", err)
+		return
+	}
+	scanner := bufio.NewScanner(bytes.NewReader(psOutput))
+	var pid int
+	for scanner.Scan() {
+		line := scanner.Text()
+		fields := strings.Fields(line)
+		if len(fields) < 2 {
+			continue
+		}
+		// Process the PID and command columns
+		pidStr := fields[0]
+		commandPath := fields[1]
+		commandName := path.Base(commandPath)
+		if len(fields) > 2 {
+			commandName = commandName + " " + strings.Join(fields[2:], " ")
+		}
+
+		if strings.HasPrefix(commandName, prefix) {
+			log.Println(commandName)
+			pid, err = strconv.Atoi(pidStr)
+			if err != nil {
+				err = fmt.Errorf("invalid pid: %s", pidStr)
+				return
+			}
+			pids = append(pids, pid)
+		}
+	}
+	return
+}
+
+func (c *IntegrationTestFramework) getTestMachineSets(ctx context.Context, namespace string) []string {
 	machineSets, err := c.ControlCluster.McmClient.MachineV1alpha1().MachineSets(namespace).List(ctx, metav1.ListOptions{})
+	testMachineSets := []string{}
 	gomega.Expect(err).NotTo(gomega.HaveOccurred())
-	count := 0
 	for _, machineSet := range machineSets.Items {
-		if machineSet.OwnerReferences[0].Name == "test-machine-deployment" {
-			count++
+		if machineSet.OwnerReferences[0].Name == testMachineDeploymentName {
+			testMachineSets = append(testMachineSets, machineSet.Name)
+		}
+	}
+	return testMachineSets
+}
+
+func (c *IntegrationTestFramework) machineSetFreezeEventCount(ctx context.Context, namespace string) int {
+	eventCount := 0
+	testMachineSets := c.getTestMachineSets(ctx, namespace)
+	machineSetFreezeReason := fmt.Sprintf("reason=%s", controller.MachineSetFreezeEvent)
+	machineSetUnfreezeReason := fmt.Sprintf("reason=%s", controller.MachineSetUnfreezeEvent)
+	for _, machineSet := range testMachineSets {
+		for _, reason := range []string{machineSetFreezeReason, machineSetUnfreezeReason} {
+			event := fmt.Sprintf("%s,involvedObject.name=%s", reason, machineSet)
+			frozenEvents, err := c.ControlCluster.Clientset.
+				CoreV1().
+				Events(namespace).
+				List(ctx, metav1.ListOptions{
+					FieldSelector: event,
+				})
+			gomega.Expect(err).NotTo(gomega.HaveOccurred())
+			eventCount += len(frozenEvents.Items)
 		}
 	}
-	return count
+	return eventCount
 }
 
 func checkMcmRepoAvailable() {
diff --git a/pkg/util/provider/machinecontroller/controller_suite_test.go b/pkg/util/provider/machinecontroller/controller_suite_test.go
index 7b6b907d2..814adddc3 100644
--- a/pkg/util/provider/machinecontroller/controller_suite_test.go
+++ b/pkg/util/provider/machinecontroller/controller_suite_test.go
@@ -605,6 +605,7 @@ func createController(
 		secretQueue:                 workqueue.NewNamedRateLimitingQueue(workqueue.DefaultControllerRateLimiter(), "secret"),
 		nodeQueue:                   workqueue.NewNamedRateLimitingQueue(workqueue.DefaultControllerRateLimiter(), "node"),
 		machineQueue:                workqueue.NewNamedRateLimitingQueue(workqueue.DefaultControllerRateLimiter(), "machine"),
+		machineTerminationQueue:     workqueue.NewNamedRateLimitingQueue(workqueue.DefaultControllerRateLimiter(), "machinetermination"),
 		machineSafetyOrphanVMsQueue: workqueue.NewNamedRateLimitingQueue(workqueue.DefaultControllerRateLimiter(), "machinesafetyorphanvms"),
 		machineSafetyAPIServerQueue: workqueue.NewNamedRateLimitingQueue(workqueue.DefaultControllerRateLimiter(), "machinesafetyapiserver"),
 		recorder:                    record.NewBroadcaster().NewRecorder(nil, corev1.EventSource{Component: ""}),
diff --git a/pkg/util/provider/machinecontroller/machine_test.go b/pkg/util/provider/machinecontroller/machine_test.go
index a658641f4..493988e87 100644
--- a/pkg/util/provider/machinecontroller/machine_test.go
+++ b/pkg/util/provider/machinecontroller/machine_test.go
@@ -226,7 +226,7 @@ var _ = Describe("machine", func() {
 				action: machineapi.Machine{
 					Spec: machineapi.MachineSpec{
 						Class: machineapi.ClassSpec{
-							Kind: "AWSMachineClass",
+							Kind: "MachineClass",
 							Name: "aws",
 						},
 					},
@@ -234,6 +234,30 @@ var _ = Describe("machine", func() {
 				expect: field.ErrorList{},
 			}),
 		)
+		DescribeTable("#machine validation fails with no class name",
+			func(data *data) {
+				errList := validation.ValidateMachine(&data.action)
+				Expect(errList).To(Equal(data.expect))
+			},
+			Entry("aws", &data{
+				action: machineapi.Machine{
+					Spec: machineapi.MachineSpec{
+						Class: machineapi.ClassSpec{
+							Kind: "MachineClass",
+							Name: "",
+						},
+					},
+				},
+				expect: field.ErrorList{
+					{
+						Type:     "FieldValueRequired",
+						Field:    "spec.class.name",
+						BadValue: "",
+						Detail:   "Name is required",
+					},
+				},
+			}),
+		)
 	})
 
 	Describe("#ValidateMachineClass", func() {