Repository: kubermatic/machine-controller
Pull request #1652 (2023-06-02 23:52:09)

[release/v1.56] Allow external CCMs to handle node objects before MC
**Manual backport of https://github.com/kubermatic/machine-controller/pull/1645**

**What this PR does / why we need it**:

Allow external CCMs to handle failing node objects before MC takes any action.
It prevents race condition between MC and CCM.

The PR introduces a `handleNodeFailuresWithExternalCCM` function that:

>handleNodeFailuresWithExternalCCM reacts to node status discovery of CCM's node lifecycle controller.
>If an instance at cloud provider is not found then it waits till CCM deletes node objects, that allows:
>   - create a new instance at cloud provider
>   - initialize a new node object - the object should not be reused between instance creation
>     for example, instance foo that got deleted and recreated should initialize a completely new node object
>     instead of reusing the old one as it can cause problems to update node's metadata, like IP address.
>
> If node is shut-down it allows MC to react accordingly to specific cloud provider requirements, those are:
>   - wait for node to become online again or
>   - delete a machine which cannot be recovered

**Which issue(s) this PR fixes**:
<!--optional, in `fixes #<issue number>` format, will close the issue(s) when PR gets merged-->
Fixes https://github.com/kubermatic/kubermatic/issues/12218

**What type of PR is this?**
/kind bug
<!--
Add one of the following kinds:
/kind bug
/kind cleanup
/kind documentation
/kind feature
/kind design

Optionally add one or more of the following kinds if applicable:
/kind api-change
/kind deprecation
/kind failing-test
/kind flake
/kind regression
/kind chore
-->

**Special notes for your reviewer**:

**Does this PR introduce a user-facing change? Then add your Release Note here**:
<!--
Write your release note:
1. Enter your extended release note in the below block. If the PR requires additional action from users switching to the new release, include the string "action required".
2. If no release note is required, just write "NONE".
-->
```release-note
Allow external CCMs to handle failing node objects before MC.
```

**Documentation**:
<!--
Please do one of the following options:
- Add a link to the existing documentation
- Add a link to the kubermatic/docs pull request
- If no documentation change is applicable then add:
  - TBD (documentation will be added later)
  - NONE (no documentation needed for this PR)
-->
```documentation
NONE
```


kubermatic-bot: [APPROVALNOTIFIER] This PR is **APPROVED**

This pull-request has been approved by: *<a href="https://github.com/kubermatic/machine-controller/pull/1652#" title="Author self-approved">mfranczy</a>*

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=kubermatic%2Fmachine-controller).

The pull request process is described [here](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process)

<details >
Needs approval from an approver in each of these files:

- ~~[OWNERS](https://github.com/kubermatic/machine-controller/blob/release/v1.56/OWNERS)~~ [mfranczy]
- ~~[pkg/cloudprovider/provider/kubevirt/OWNERS](https://github.com/kubermatic/machine-controller/blob/release/v1.56/pkg/cloudprovider/provider/kubevirt/OWNERS)~~ [mfranczy]

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":[]} -->
mfranczy: /test pull-machine-controller-e2e-kubevirt
kubermatic-bot: @mfranczy: The following test **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-machine-controller-build | 9723307f8879803a7ff143c791ee4d47411d4d94 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1652/pull-machine-controller-build/1664782002387161088) | true | `/test pull-machine-controller-build`

[Full PR test history](https://public-prow.loodse.com/pr-history?org=kubermatic&repo=machine-controller&pr=1652)

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
kubermatic-bot: @mfranczy: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-machine-controller-build | 9723307f8879803a7ff143c791ee4d47411d4d94 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1652/pull-machine-controller-build/1664782002387161088) | true | `/test pull-machine-controller-build`
pull-machine-controller-e2e-aws | 9723307f8879803a7ff143c791ee4d47411d4d94 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1652/pull-machine-controller-e2e-aws/1664782002156474368) | true | `/test pull-machine-controller-e2e-aws`

[Full PR test history](https://public-prow.loodse.com/pr-history?org=kubermatic&repo=machine-controller&pr=1652)

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
kubermatic-bot: @mfranczy: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-machine-controller-build | 9723307f8879803a7ff143c791ee4d47411d4d94 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1652/pull-machine-controller-build/1664782002387161088) | true | `/test pull-machine-controller-build`
pull-machine-controller-e2e-aws | 9723307f8879803a7ff143c791ee4d47411d4d94 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1652/pull-machine-controller-e2e-aws/1664782002156474368) | true | `/test pull-machine-controller-e2e-aws`
pull-machine-controller-e2e-ubuntu-upgrade | 9723307f8879803a7ff143c791ee4d47411d4d94 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1652/pull-machine-controller-e2e-ubuntu-upgrade/1664782002101948416) | true | `/test pull-machine-controller-e2e-ubuntu-upgrade`

[Full PR test history](https://public-prow.loodse.com/pr-history?org=kubermatic&repo=machine-controller&pr=1652)

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
kubermatic-bot: @mfranczy: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-machine-controller-build | 9723307f8879803a7ff143c791ee4d47411d4d94 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1652/pull-machine-controller-build/1664782002387161088) | true | `/test pull-machine-controller-build`
pull-machine-controller-e2e-aws | 9723307f8879803a7ff143c791ee4d47411d4d94 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1652/pull-machine-controller-e2e-aws/1664782002156474368) | true | `/test pull-machine-controller-e2e-aws`
pull-machine-controller-e2e-ubuntu-upgrade | 9723307f8879803a7ff143c791ee4d47411d4d94 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1652/pull-machine-controller-e2e-ubuntu-upgrade/1664782002101948416) | true | `/test pull-machine-controller-e2e-ubuntu-upgrade`
pull-machine-controller-license-validation | 9723307f8879803a7ff143c791ee4d47411d4d94 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1652/pull-machine-controller-license-validation/1664782005239287808) | true | `/test pull-machine-controller-license-validation`

[Full PR test history](https://public-prow.loodse.com/pr-history?org=kubermatic&repo=machine-controller&pr=1652)

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
kubermatic-bot: @mfranczy: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-machine-controller-build | 9723307f8879803a7ff143c791ee4d47411d4d94 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1652/pull-machine-controller-build/1664782002387161088) | true | `/test pull-machine-controller-build`
pull-machine-controller-e2e-aws | 9723307f8879803a7ff143c791ee4d47411d4d94 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1652/pull-machine-controller-e2e-aws/1664782002156474368) | true | `/test pull-machine-controller-e2e-aws`
pull-machine-controller-e2e-ubuntu-upgrade | 9723307f8879803a7ff143c791ee4d47411d4d94 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1652/pull-machine-controller-e2e-ubuntu-upgrade/1664782002101948416) | true | `/test pull-machine-controller-e2e-ubuntu-upgrade`
pull-machine-controller-license-validation | 9723307f8879803a7ff143c791ee4d47411d4d94 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1652/pull-machine-controller-license-validation/1664782005239287808) | true | `/test pull-machine-controller-license-validation`
pull-machine-controller-e2e-aws-legacy-userdata | 9723307f8879803a7ff143c791ee4d47411d4d94 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1652/pull-machine-controller-e2e-aws-legacy-userdata/1664782002227777536) | true | `/test pull-machine-controller-e2e-aws-legacy-userdata`

[Full PR test history](https://public-prow.loodse.com/pr-history?org=kubermatic&repo=machine-controller&pr=1652)

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
kubermatic-bot: @mfranczy: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-machine-controller-build | 9723307f8879803a7ff143c791ee4d47411d4d94 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1652/pull-machine-controller-build/1664782002387161088) | true | `/test pull-machine-controller-build`
pull-machine-controller-e2e-aws | 9723307f8879803a7ff143c791ee4d47411d4d94 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1652/pull-machine-controller-e2e-aws/1664782002156474368) | true | `/test pull-machine-controller-e2e-aws`
pull-machine-controller-e2e-ubuntu-upgrade | 9723307f8879803a7ff143c791ee4d47411d4d94 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1652/pull-machine-controller-e2e-ubuntu-upgrade/1664782002101948416) | true | `/test pull-machine-controller-e2e-ubuntu-upgrade`
pull-machine-controller-license-validation | 9723307f8879803a7ff143c791ee4d47411d4d94 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1652/pull-machine-controller-license-validation/1664782005239287808) | true | `/test pull-machine-controller-license-validation`
pull-machine-controller-e2e-aws-legacy-userdata | 9723307f8879803a7ff143c791ee4d47411d4d94 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1652/pull-machine-controller-e2e-aws-legacy-userdata/1664782002227777536) | true | `/test pull-machine-controller-e2e-aws-legacy-userdata`
pull-machine-controller-e2e-invalid-objects-get-rejected | 9723307f8879803a7ff143c791ee4d47411d4d94 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1652/pull-machine-controller-e2e-invalid-objects-get-rejected/1664782001988702208) | true | `/test pull-machine-controller-e2e-invalid-objects-get-rejected`

[Full PR test history](https://public-prow.loodse.com/pr-history?org=kubermatic&repo=machine-controller&pr=1652)

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
kubermatic-bot: @mfranczy: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-machine-controller-build | 9723307f8879803a7ff143c791ee4d47411d4d94 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1652/pull-machine-controller-build/1664782002387161088) | true | `/test pull-machine-controller-build`
pull-machine-controller-e2e-aws | 9723307f8879803a7ff143c791ee4d47411d4d94 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1652/pull-machine-controller-e2e-aws/1664782002156474368) | true | `/test pull-machine-controller-e2e-aws`
pull-machine-controller-e2e-ubuntu-upgrade | 9723307f8879803a7ff143c791ee4d47411d4d94 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1652/pull-machine-controller-e2e-ubuntu-upgrade/1664782002101948416) | true | `/test pull-machine-controller-e2e-ubuntu-upgrade`
pull-machine-controller-license-validation | 9723307f8879803a7ff143c791ee4d47411d4d94 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1652/pull-machine-controller-license-validation/1664782005239287808) | true | `/test pull-machine-controller-license-validation`
pull-machine-controller-e2e-aws-legacy-userdata | 9723307f8879803a7ff143c791ee4d47411d4d94 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1652/pull-machine-controller-e2e-aws-legacy-userdata/1664782002227777536) | true | `/test pull-machine-controller-e2e-aws-legacy-userdata`
pull-machine-controller-e2e-invalid-objects-get-rejected | 9723307f8879803a7ff143c791ee4d47411d4d94 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1652/pull-machine-controller-e2e-invalid-objects-get-rejected/1664782001988702208) | true | `/test pull-machine-controller-e2e-invalid-objects-get-rejected`
pull-machine-controller-e2e-azure | 9723307f8879803a7ff143c791ee4d47411d4d94 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1652/pull-machine-controller-e2e-azure/1664782002286497792) | true | `/test pull-machine-controller-e2e-azure`

[Full PR test history](https://public-prow.loodse.com/pr-history?org=kubermatic&repo=machine-controller&pr=1652)

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
kubermatic-bot: @mfranczy: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-machine-controller-build | 9723307f8879803a7ff143c791ee4d47411d4d94 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1652/pull-machine-controller-build/1664782002387161088) | true | `/test pull-machine-controller-build`
pull-machine-controller-e2e-aws | 9723307f8879803a7ff143c791ee4d47411d4d94 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1652/pull-machine-controller-e2e-aws/1664782002156474368) | true | `/test pull-machine-controller-e2e-aws`
pull-machine-controller-e2e-ubuntu-upgrade | 9723307f8879803a7ff143c791ee4d47411d4d94 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1652/pull-machine-controller-e2e-ubuntu-upgrade/1664782002101948416) | true | `/test pull-machine-controller-e2e-ubuntu-upgrade`
pull-machine-controller-license-validation | 9723307f8879803a7ff143c791ee4d47411d4d94 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1652/pull-machine-controller-license-validation/1664782005239287808) | true | `/test pull-machine-controller-license-validation`
pull-machine-controller-e2e-aws-legacy-userdata | 9723307f8879803a7ff143c791ee4d47411d4d94 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1652/pull-machine-controller-e2e-aws-legacy-userdata/1664782002227777536) | true | `/test pull-machine-controller-e2e-aws-legacy-userdata`
pull-machine-controller-e2e-invalid-objects-get-rejected | 9723307f8879803a7ff143c791ee4d47411d4d94 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1652/pull-machine-controller-e2e-invalid-objects-get-rejected/1664782001988702208) | true | `/test pull-machine-controller-e2e-invalid-objects-get-rejected`
pull-machine-controller-e2e-azure | 9723307f8879803a7ff143c791ee4d47411d4d94 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1652/pull-machine-controller-e2e-azure/1664782002286497792) | true | `/test pull-machine-controller-e2e-azure`
pull-machine-controller-e2e-custom-ca | 9723307f8879803a7ff143c791ee4d47411d4d94 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1652/pull-machine-controller-e2e-custom-ca/1664782002043228160) | true | `/test pull-machine-controller-e2e-custom-ca`

[Full PR test history](https://public-prow.loodse.com/pr-history?org=kubermatic&repo=machine-controller&pr=1652)

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
kubermatic-bot: @mfranczy: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-machine-controller-build | 9723307f8879803a7ff143c791ee4d47411d4d94 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1652/pull-machine-controller-build/1664782002387161088) | true | `/test pull-machine-controller-build`
pull-machine-controller-e2e-aws | 9723307f8879803a7ff143c791ee4d47411d4d94 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1652/pull-machine-controller-e2e-aws/1664782002156474368) | true | `/test pull-machine-controller-e2e-aws`
pull-machine-controller-e2e-ubuntu-upgrade | 9723307f8879803a7ff143c791ee4d47411d4d94 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1652/pull-machine-controller-e2e-ubuntu-upgrade/1664782002101948416) | true | `/test pull-machine-controller-e2e-ubuntu-upgrade`
pull-machine-controller-license-validation | 9723307f8879803a7ff143c791ee4d47411d4d94 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1652/pull-machine-controller-license-validation/1664782005239287808) | true | `/test pull-machine-controller-license-validation`
pull-machine-controller-e2e-aws-legacy-userdata | 9723307f8879803a7ff143c791ee4d47411d4d94 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1652/pull-machine-controller-e2e-aws-legacy-userdata/1664782002227777536) | true | `/test pull-machine-controller-e2e-aws-legacy-userdata`
pull-machine-controller-e2e-invalid-objects-get-rejected | 9723307f8879803a7ff143c791ee4d47411d4d94 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1652/pull-machine-controller-e2e-invalid-objects-get-rejected/1664782001988702208) | true | `/test pull-machine-controller-e2e-invalid-objects-get-rejected`
pull-machine-controller-e2e-azure | 9723307f8879803a7ff143c791ee4d47411d4d94 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1652/pull-machine-controller-e2e-azure/1664782002286497792) | true | `/test pull-machine-controller-e2e-azure`
pull-machine-controller-e2e-custom-ca | 9723307f8879803a7ff143c791ee4d47411d4d94 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1652/pull-machine-controller-e2e-custom-ca/1664782002043228160) | true | `/test pull-machine-controller-e2e-custom-ca`
pull-machine-controller-lint | 9723307f8879803a7ff143c791ee4d47411d4d94 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1652/pull-machine-controller-lint/1664782002731094016) | true | `/test pull-machine-controller-lint`

[Full PR test history](https://public-prow.loodse.com/pr-history?org=kubermatic&repo=machine-controller&pr=1652)

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
kubermatic-bot: @mfranczy: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-machine-controller-build | 9723307f8879803a7ff143c791ee4d47411d4d94 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1652/pull-machine-controller-build/1664782002387161088) | true | `/test pull-machine-controller-build`
pull-machine-controller-e2e-aws | 9723307f8879803a7ff143c791ee4d47411d4d94 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1652/pull-machine-controller-e2e-aws/1664782002156474368) | true | `/test pull-machine-controller-e2e-aws`
pull-machine-controller-e2e-ubuntu-upgrade | 9723307f8879803a7ff143c791ee4d47411d4d94 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1652/pull-machine-controller-e2e-ubuntu-upgrade/1664782002101948416) | true | `/test pull-machine-controller-e2e-ubuntu-upgrade`
pull-machine-controller-license-validation | 9723307f8879803a7ff143c791ee4d47411d4d94 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1652/pull-machine-controller-license-validation/1664782005239287808) | true | `/test pull-machine-controller-license-validation`
pull-machine-controller-e2e-aws-legacy-userdata | 9723307f8879803a7ff143c791ee4d47411d4d94 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1652/pull-machine-controller-e2e-aws-legacy-userdata/1664782002227777536) | true | `/test pull-machine-controller-e2e-aws-legacy-userdata`
pull-machine-controller-e2e-invalid-objects-get-rejected | 9723307f8879803a7ff143c791ee4d47411d4d94 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1652/pull-machine-controller-e2e-invalid-objects-get-rejected/1664782001988702208) | true | `/test pull-machine-controller-e2e-invalid-objects-get-rejected`
pull-machine-controller-e2e-azure | 9723307f8879803a7ff143c791ee4d47411d4d94 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1652/pull-machine-controller-e2e-azure/1664782002286497792) | true | `/test pull-machine-controller-e2e-azure`
pull-machine-controller-e2e-custom-ca | 9723307f8879803a7ff143c791ee4d47411d4d94 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1652/pull-machine-controller-e2e-custom-ca/1664782002043228160) | true | `/test pull-machine-controller-e2e-custom-ca`
pull-machine-controller-lint | 9723307f8879803a7ff143c791ee4d47411d4d94 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1652/pull-machine-controller-lint/1664782002731094016) | true | `/test pull-machine-controller-lint`
pull-machine-controller-test | 9723307f8879803a7ff143c791ee4d47411d4d94 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1652/pull-machine-controller-test/1664782006073954304) | true | `/test pull-machine-controller-test`

[Full PR test history](https://public-prow.loodse.com/pr-history?org=kubermatic&repo=machine-controller&pr=1652)

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
mfranczy: /retest
kubermatic-bot: @mfranczy: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-machine-controller-e2e-aws | 9723307f8879803a7ff143c791ee4d47411d4d94 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1652/pull-machine-controller-e2e-aws/1664782002156474368) | true | `/test pull-machine-controller-e2e-aws`
pull-machine-controller-e2e-ubuntu-upgrade | 9723307f8879803a7ff143c791ee4d47411d4d94 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1652/pull-machine-controller-e2e-ubuntu-upgrade/1664782002101948416) | true | `/test pull-machine-controller-e2e-ubuntu-upgrade`
pull-machine-controller-e2e-aws-legacy-userdata | 9723307f8879803a7ff143c791ee4d47411d4d94 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1652/pull-machine-controller-e2e-aws-legacy-userdata/1664782002227777536) | true | `/test pull-machine-controller-e2e-aws-legacy-userdata`
pull-machine-controller-e2e-invalid-objects-get-rejected | 9723307f8879803a7ff143c791ee4d47411d4d94 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1652/pull-machine-controller-e2e-invalid-objects-get-rejected/1664782001988702208) | true | `/test pull-machine-controller-e2e-invalid-objects-get-rejected`
pull-machine-controller-e2e-azure | 9723307f8879803a7ff143c791ee4d47411d4d94 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1652/pull-machine-controller-e2e-azure/1664782002286497792) | true | `/test pull-machine-controller-e2e-azure`
pull-machine-controller-e2e-custom-ca | 9723307f8879803a7ff143c791ee4d47411d4d94 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1652/pull-machine-controller-e2e-custom-ca/1664782002043228160) | true | `/test pull-machine-controller-e2e-custom-ca`
pull-machine-controller-lint | 9723307f8879803a7ff143c791ee4d47411d4d94 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1652/pull-machine-controller-lint/1664782002731094016) | true | `/test pull-machine-controller-lint`
pull-machine-controller-test | 9723307f8879803a7ff143c791ee4d47411d4d94 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1652/pull-machine-controller-test/1664783277992448000) | true | `/test pull-machine-controller-test`

[Full PR test history](https://public-prow.loodse.com/pr-history?org=kubermatic&repo=machine-controller&pr=1652)

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
kubermatic-bot: @mfranczy: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-machine-controller-e2e-aws | 9723307f8879803a7ff143c791ee4d47411d4d94 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1652/pull-machine-controller-e2e-aws/1664782002156474368) | true | `/test pull-machine-controller-e2e-aws`
pull-machine-controller-e2e-ubuntu-upgrade | 9723307f8879803a7ff143c791ee4d47411d4d94 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1652/pull-machine-controller-e2e-ubuntu-upgrade/1664782002101948416) | true | `/test pull-machine-controller-e2e-ubuntu-upgrade`
pull-machine-controller-e2e-aws-legacy-userdata | 9723307f8879803a7ff143c791ee4d47411d4d94 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1652/pull-machine-controller-e2e-aws-legacy-userdata/1664782002227777536) | true | `/test pull-machine-controller-e2e-aws-legacy-userdata`
pull-machine-controller-e2e-invalid-objects-get-rejected | 9723307f8879803a7ff143c791ee4d47411d4d94 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1652/pull-machine-controller-e2e-invalid-objects-get-rejected/1664782001988702208) | true | `/test pull-machine-controller-e2e-invalid-objects-get-rejected`
pull-machine-controller-e2e-azure | 9723307f8879803a7ff143c791ee4d47411d4d94 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1652/pull-machine-controller-e2e-azure/1664782002286497792) | true | `/test pull-machine-controller-e2e-azure`
pull-machine-controller-e2e-custom-ca | 9723307f8879803a7ff143c791ee4d47411d4d94 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1652/pull-machine-controller-e2e-custom-ca/1664782002043228160) | true | `/test pull-machine-controller-e2e-custom-ca`
pull-machine-controller-test | 9723307f8879803a7ff143c791ee4d47411d4d94 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1652/pull-machine-controller-test/1664783277992448000) | true | `/test pull-machine-controller-test`
pull-machine-controller-lint | 9723307f8879803a7ff143c791ee4d47411d4d94 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1652/pull-machine-controller-lint/1664783277891784704) | true | `/test pull-machine-controller-lint`

[Full PR test history](https://public-prow.loodse.com/pr-history?org=kubermatic&repo=machine-controller&pr=1652)

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
mfranczy: /test pull-machine-controller-e2e-gce
kubermatic-bot: LGTM label has been added.  <details>Git tree hash: b1fcc5a94ebdd8353b55d9d2a4d8fd7bab71b0c6</details>

Patch:
diff --git a/go.mod b/go.mod
index 3a754384..7b0f2341 100644
--- a/go.mod
+++ b/go.mod
@@ -52,6 +52,7 @@ require (
 	k8s.io/apiextensions-apiserver v0.26.0
 	k8s.io/apimachinery v0.26.0
 	k8s.io/client-go v12.0.0+incompatible
+	k8s.io/cloud-provider v0.26.0
 	k8s.io/klog v1.0.0
 	k8s.io/kubelet v0.26.0
 	k8s.io/utils v0.0.0-20221128185143-99ec85e7a448
diff --git a/go.sum b/go.sum
index 2eb99649..b0949b3a 100644
--- a/go.sum
+++ b/go.sum
@@ -1192,6 +1192,8 @@ k8s.io/apimachinery v0.26.0 h1:1feANjElT7MvPqp0JT6F3Ss6TWDwmcjLypwoPpEf7zg=
 k8s.io/apimachinery v0.26.0/go.mod h1:tnPmbONNJ7ByJNz9+n9kMjNP8ON+1qoAIIC70lztu74=
 k8s.io/client-go v0.26.0 h1:lT1D3OfO+wIi9UFolCrifbjUUgu7CpLca0AD8ghRLI8=
 k8s.io/client-go v0.26.0/go.mod h1:I2Sh57A79EQsDmn7F7ASpmru1cceh3ocVT9KlX2jEZg=
+k8s.io/cloud-provider v0.26.0 h1:kO2BIgCou71QNRHGkpFi/8lnas9UIr+fJz1l/nuiOMo=
+k8s.io/cloud-provider v0.26.0/go.mod h1:JwfUAH67C8f7t6tOC4v4ty+DuvIYVjNF6bGVYSDCqqs=
 k8s.io/code-generator v0.23.0/go.mod h1:vQvOhDXhuzqiVfM/YHp+dmg10WDZCchJVObc9MvowsE=
 k8s.io/code-generator v0.23.3/go.mod h1:S0Q1JVA+kSzTI1oUvbKAxZY/DYbA/ZUb4Uknog12ETk=
 k8s.io/component-base v0.26.0 h1:0IkChOCohtDHttmKuz+EP3j3+qKmV55rM9gIFTXA7Vs=
diff --git a/pkg/cloudprovider/provider/kubevirt/provider.go b/pkg/cloudprovider/provider/kubevirt/provider.go
index 044439b1..85e71709 100644
--- a/pkg/cloudprovider/provider/kubevirt/provider.go
+++ b/pkg/cloudprovider/provider/kubevirt/provider.go
@@ -445,18 +445,6 @@ func (p *provider) Get(ctx context.Context, machine *clusterv1alpha1.Machine, _
 		return nil, cloudprovidererrors.ErrInstanceNotFound
 	}
 
-	if virtualMachineInstance.Status.Phase == kubevirtv1.Failed ||
-		// The VMI enters phase succeeded if someone issues a kubectl
-		// delete pod on the virt-launcher pod it runs in
-		virtualMachineInstance.Status.Phase == kubevirtv1.Succeeded {
-		// The pod got deleted, delete the VMI and return ErrNotFound so the VMI
-		// will get recreated
-		if err := sigClient.Delete(ctx, virtualMachineInstance); err != nil {
-			return nil, fmt.Errorf("failed to delete failed VMI %s: %w", machine.Name, err)
-		}
-		return nil, cloudprovidererrors.ErrInstanceNotFound
-	}
-
 	return &kubeVirtServer{vmi: *virtualMachineInstance}, nil
 }
 
@@ -633,6 +621,8 @@ func (p *provider) newVirtualMachine(ctx context.Context, c *Config, pc *provide
 		return nil, fmt.Errorf("could not compute a random MAC address")
 	}
 
+	runStrategyOnce := kubevirtv1.RunStrategyOnce
+
 	virtualMachine := &kubevirtv1.VirtualMachine{
 		ObjectMeta: metav1.ObjectMeta{
 			Name:      machine.Name,
@@ -640,7 +630,7 @@ func (p *provider) newVirtualMachine(ctx context.Context, c *Config, pc *provide
 			Labels:    labels,
 		},
 		Spec: kubevirtv1.VirtualMachineSpec{
-			Running:      utilpointer.Bool(true),
+			RunStrategy:  &runStrategyOnce,
 			Instancetype: c.Instancetype,
 			Preference:   c.Preference,
 			Template: &kubevirtv1.VirtualMachineInstanceTemplateSpec{
@@ -692,7 +682,6 @@ func (p *provider) Cleanup(ctx context.Context, machine *clusterv1alpha1.Machine
 		if !kerrors.IsNotFound(err) {
 			return false, fmt.Errorf("failed to get VirtualMachineInstance %s: %w", machine.Name, err)
 		}
-		// VMI is gone
 		return true, nil
 	}
 
diff --git a/pkg/controller/machine/machine_controller.go b/pkg/controller/machine/machine_controller.go
index d5f0f47e..49980146 100644
--- a/pkg/controller/machine/machine_controller.go
+++ b/pkg/controller/machine/machine_controller.go
@@ -63,6 +63,7 @@ import (
 	"k8s.io/client-go/tools/record"
 	"k8s.io/client-go/tools/reference"
 	"k8s.io/client-go/util/retry"
+	ccmapi "k8s.io/cloud-provider/api"
 	"k8s.io/klog"
 	"sigs.k8s.io/controller-runtime/pkg/client"
 	ctrlruntimeclient "sigs.k8s.io/controller-runtime/pkg/client"
@@ -411,7 +412,8 @@ func (r *Reconciler) reconcile(ctx context.Context, machine *clusterv1alpha1.Mac
 
 	// step 2: check if a user requested to delete the machine
 	if machine.DeletionTimestamp != nil {
-		return r.deleteMachine(ctx, prov, providerConfig.CloudProvider, machine)
+		skipEviction := false
+		return r.deleteMachine(ctx, prov, providerConfig.CloudProvider, machine, skipEviction)
 	}
 
 	// Step 3: Essentially creates an instance for the given machine.
@@ -445,6 +447,9 @@ func (r *Reconciler) reconcile(ctx context.Context, machine *clusterv1alpha1.Mac
 		}
 	} else {
 		// Node is not ready anymore? Maybe it got deleted
+		if r.nodeSettings.ExternalCloudProvider {
+			return r.handleNodeFailuresWithExternalCCM(ctx, prov, providerConfig, node, machine)
+		}
 		return r.ensureInstanceExistsForMachine(ctx, prov, machine, userdataPlugin, providerConfig)
 	}
 
@@ -493,7 +498,7 @@ func (r *Reconciler) shouldCleanupVolumes(ctx context.Context, machine *clusterv
 func (r *Reconciler) shouldEvict(ctx context.Context, machine *clusterv1alpha1.Machine) (bool, error) {
 	// If the deletion got triggered a few hours ago, skip eviction.
 	// We assume here that the eviction is blocked by misconfiguration or a misbehaving kubelet and/or controller-runtime
-	if time.Since(machine.DeletionTimestamp.Time) > r.skipEvictionAfter {
+	if machine.DeletionTimestamp != nil && time.Since(machine.DeletionTimestamp.Time) > r.skipEvictionAfter {
 		klog.V(0).Infof("Skipping eviction for machine %q since the deletion got triggered %.2f minutes ago", machine.Name, r.skipEvictionAfter.Minutes())
 		return false, nil
 	}
@@ -548,11 +553,25 @@ func (r *Reconciler) shouldEvict(ctx context.Context, machine *clusterv1alpha1.M
 }
 
 // deleteMachine makes sure that an instance has gone in a series of steps.
-func (r *Reconciler) deleteMachine(ctx context.Context, prov cloudprovidertypes.Provider, providerName providerconfigtypes.CloudProvider, machine *clusterv1alpha1.Machine) (*reconcile.Result, error) {
-	shouldEvict, err := r.shouldEvict(ctx, machine)
-	if err != nil {
-		return nil, err
+func (r *Reconciler) deleteMachine(
+	ctx context.Context,
+	prov cloudprovidertypes.Provider,
+	providerName providerconfigtypes.CloudProvider,
+	machine *clusterv1alpha1.Machine,
+	skipEviction bool,
+) (*reconcile.Result, error) {
+	var (
+		shouldEvict bool
+		err         error
+	)
+
+	if !skipEviction {
+		shouldEvict, err = r.shouldEvict(ctx, machine)
+		if err != nil {
+			return nil, err
+		}
 	}
+
 	shouldCleanUpVolumes, err := r.shouldCleanupVolumes(ctx, machine, providerName)
 	if err != nil {
 		return nil, err
@@ -650,7 +669,6 @@ func (r *Reconciler) deleteCloudProviderInstance(ctx context.Context, prov cloud
 		message := fmt.Sprintf("%v. Please manually delete %s finalizer from the machine object.", err, FinalizerDeleteInstance)
 		return nil, r.updateMachineErrorIfTerminalError(machine, common.DeleteMachineError, message, err, "failed to delete machine at cloud provider")
 	}
-
 	if !completelyGone {
 		// As the instance is not completely gone yet, we need to recheck in a few seconds.
 		return &reconcile.Result{RequeueAfter: deletionRetryWaitPeriod}, nil
@@ -995,14 +1013,6 @@ func (r *Reconciler) ensureNodeLabelsAnnotationsAndTaints(ctx context.Context, n
 		modifiers = append(modifiers, f(AnnotationAutoscalerIdentifier, autoscalerAnnotationValue))
 	}
 
-	taintExists := func(node *corev1.Node, taint corev1.Taint) bool {
-		for _, t := range node.Spec.Taints {
-			if t.MatchTaint(&taint) {
-				return true
-			}
-		}
-		return false
-	}
 	for _, t := range machine.Spec.Taints {
 		if !taintExists(node, t) {
 			f := func(t corev1.Taint) func(*corev1.Node) {
@@ -1125,6 +1135,15 @@ func findNodeByProviderID(instance instance.Instance, provider providerconfigtyp
 	return nil
 }
 
+func taintExists(node *corev1.Node, taint corev1.Taint) bool {
+	for _, t := range node.Spec.Taints {
+		if t.MatchTaint(&taint) {
+			return true
+		}
+	}
+	return false
+}
+
 func (r *Reconciler) ReadinessChecks(ctx context.Context) map[string]healthcheck.Check {
 	return map[string]healthcheck.Check{
 		"valid-info-kubeconfig": func() error {
@@ -1184,3 +1203,45 @@ func (r *Reconciler) updateNode(ctx context.Context, node *corev1.Node, modifier
 		return r.client.Update(ctx, node)
 	})
 }
+
+// handleNodeFailuresWithExternalCCM reacts to node status discovery of CCM's node lifecycle controller.
+// If an instance at cloud provider is not found then it waits till CCM deletes node objects, that allows:
+//   - create a new instance at cloud provider
+//   - initialize a new node object - the object should not be reused between instance creation
+//     for example, instance foo that got deleted and recreated should initialize a completely new node object
+//     instead of reusing the old one as it can cause problems to update node's metadata, like IP address.
+//
+// If node is shut-down it allows MC to react accordingly to specific cloud provider requirements, those are:
+//   - wait for node to become online again or
+//   - delete a machine which cannot be recovered
+func (r *Reconciler) handleNodeFailuresWithExternalCCM(
+	ctx context.Context,
+	prov cloudprovidertypes.Provider,
+	provConfig *providerconfigtypes.Config,
+	node *corev1.Node,
+	machine *clusterv1alpha1.Machine,
+) (*reconcile.Result, error) {
+	taintShutdown := corev1.Taint{
+		Key:    ccmapi.TaintNodeShutdown,
+		Effect: corev1.TaintEffectNoSchedule,
+	}
+
+	_, err := prov.Get(ctx, machine, r.providerData)
+	if err != nil {
+		if cloudprovidererrors.IsNotFound(err) {
+			klog.V(0).Infof("The node %q does not have corresponding instance, waiting for CCM to delete it", node.Name)
+			return &reconcile.Result{RequeueAfter: deletionRetryWaitPeriod}, nil
+		}
+		return nil, err
+	} else if taintExists(node, taintShutdown) {
+		switch provConfig.CloudProvider {
+		case providerconfigtypes.CloudProviderKubeVirt:
+			klog.V(0).Infof("Deleting a shut-down machine %q that cannot recover", machine.Name)
+			skipEviction := true
+			return r.deleteMachine(ctx, prov, providerconfigtypes.CloudProviderKubeVirt, machine, skipEviction)
+		}
+	}
+
+	klog.V(4).Infof("Waiting for a node to become %q", corev1.NodeReady)
+	return &reconcile.Result{RequeueAfter: deletionRetryWaitPeriod}, err
+}


Test patch:
diff --git a/pkg/cloudprovider/provider/kubevirt/testdata/affinity-no-values.yaml b/pkg/cloudprovider/provider/kubevirt/testdata/affinity-no-values.yaml
index f304b2a6..7d01a42c 100644
--- a/pkg/cloudprovider/provider/kubevirt/testdata/affinity-no-values.yaml
+++ b/pkg/cloudprovider/provider/kubevirt/testdata/affinity-no-values.yaml
@@ -25,7 +25,7 @@ spec:
         source:
           http:
             url: http://x.y.z.t/ubuntu.img
-  running: true
+  runStrategy: Once
   template:
     metadata:
       labels:
diff --git a/pkg/cloudprovider/provider/kubevirt/testdata/affinity.yaml b/pkg/cloudprovider/provider/kubevirt/testdata/affinity.yaml
index 309e30d7..a28aded5 100644
--- a/pkg/cloudprovider/provider/kubevirt/testdata/affinity.yaml
+++ b/pkg/cloudprovider/provider/kubevirt/testdata/affinity.yaml
@@ -25,7 +25,7 @@ spec:
         source:
           http:
             url: http://x.y.z.t/ubuntu.img
-  running: true
+  runStrategy: Once
   template:
     metadata:
       labels:
diff --git a/pkg/cloudprovider/provider/kubevirt/testdata/custom-local-disk.yaml b/pkg/cloudprovider/provider/kubevirt/testdata/custom-local-disk.yaml
index e10d07c6..b77494b6 100644
--- a/pkg/cloudprovider/provider/kubevirt/testdata/custom-local-disk.yaml
+++ b/pkg/cloudprovider/provider/kubevirt/testdata/custom-local-disk.yaml
@@ -25,7 +25,7 @@ spec:
           pvc:
             namespace: ns
             name: dvname
-  running: true
+  runStrategy: Once
   template:
     metadata:
       creationTimestamp: null
diff --git a/pkg/cloudprovider/provider/kubevirt/testdata/http-image-source.yaml b/pkg/cloudprovider/provider/kubevirt/testdata/http-image-source.yaml
index 3bf7731e..caef1ed5 100644
--- a/pkg/cloudprovider/provider/kubevirt/testdata/http-image-source.yaml
+++ b/pkg/cloudprovider/provider/kubevirt/testdata/http-image-source.yaml
@@ -24,7 +24,7 @@ spec:
         source:
           http:
             url: http://x.y.z.t/ubuntu.img
-  running: true
+  runStrategy: Once
   template:
     metadata:
       creationTimestamp: null
diff --git a/pkg/cloudprovider/provider/kubevirt/testdata/instancetype-preference-custom.yaml b/pkg/cloudprovider/provider/kubevirt/testdata/instancetype-preference-custom.yaml
index 060171c4..ba699302 100644
--- a/pkg/cloudprovider/provider/kubevirt/testdata/instancetype-preference-custom.yaml
+++ b/pkg/cloudprovider/provider/kubevirt/testdata/instancetype-preference-custom.yaml
@@ -25,7 +25,7 @@ spec:
         source:
           http:
             url: http://x.y.z.t/ubuntu.img
-  running: true
+  runStrategy: Once
   instancetype:
     kind: VirtualMachineClusterInstancetype
     name: custom-it
diff --git a/pkg/cloudprovider/provider/kubevirt/testdata/instancetype-preference-standard.yaml b/pkg/cloudprovider/provider/kubevirt/testdata/instancetype-preference-standard.yaml
index b56229f6..1f54c87a 100644
--- a/pkg/cloudprovider/provider/kubevirt/testdata/instancetype-preference-standard.yaml
+++ b/pkg/cloudprovider/provider/kubevirt/testdata/instancetype-preference-standard.yaml
@@ -24,7 +24,7 @@ spec:
         source:
           http:
             url: http://x.y.z.t/ubuntu.img
-  running: true
+  runStrategy: Once
   instancetype:
     kind: VirtualMachineInstancetype
     name: standard-it
diff --git a/pkg/cloudprovider/provider/kubevirt/testdata/nominal-case.yaml b/pkg/cloudprovider/provider/kubevirt/testdata/nominal-case.yaml
index 5dddb4b5..24afa309 100644
--- a/pkg/cloudprovider/provider/kubevirt/testdata/nominal-case.yaml
+++ b/pkg/cloudprovider/provider/kubevirt/testdata/nominal-case.yaml
@@ -24,7 +24,7 @@ spec:
         source:
           http:
             url: http://x.y.z.t/ubuntu.img
-  running: true
+  runStrategy: Once
   template:
     metadata:
       creationTimestamp: null
diff --git a/pkg/cloudprovider/provider/kubevirt/testdata/pvc-image-source.yaml b/pkg/cloudprovider/provider/kubevirt/testdata/pvc-image-source.yaml
index 2cafabbd..978213ec 100644
--- a/pkg/cloudprovider/provider/kubevirt/testdata/pvc-image-source.yaml
+++ b/pkg/cloudprovider/provider/kubevirt/testdata/pvc-image-source.yaml
@@ -25,7 +25,7 @@ spec:
           pvc:
             namespace: ns
             name: dvname
-  running: true
+  runStrategy: Once
   template:
     metadata:
       creationTimestamp: null
diff --git a/pkg/cloudprovider/provider/kubevirt/testdata/secondary-disks.yaml b/pkg/cloudprovider/provider/kubevirt/testdata/secondary-disks.yaml
index 576172c9..0ce57a4b 100644
--- a/pkg/cloudprovider/provider/kubevirt/testdata/secondary-disks.yaml
+++ b/pkg/cloudprovider/provider/kubevirt/testdata/secondary-disks.yaml
@@ -50,7 +50,7 @@ spec:
         source:
           http:
             url: http://x.y.z.t/ubuntu.img
-  running: true
+  runStrategy: Once
   template:
     metadata:
       creationTimestamp: null
diff --git a/pkg/cloudprovider/provider/kubevirt/testdata/topologyspreadconstraints.yaml b/pkg/cloudprovider/provider/kubevirt/testdata/topologyspreadconstraints.yaml
index a36f6b84..36346072 100644
--- a/pkg/cloudprovider/provider/kubevirt/testdata/topologyspreadconstraints.yaml
+++ b/pkg/cloudprovider/provider/kubevirt/testdata/topologyspreadconstraints.yaml
@@ -24,7 +24,7 @@ spec:
         source:
           http:
             url: http://x.y.z.t/ubuntu.img
-  running: true
+  runStrategy: Once
   template:
     metadata:
       creationTimestamp: null
diff --git a/test/e2e/provisioning/testdata/machinedeployment-kubevirt.yaml b/test/e2e/provisioning/testdata/machinedeployment-kubevirt.yaml
index 90a46bfc..d97169ad 100644
--- a/test/e2e/provisioning/testdata/machinedeployment-kubevirt.yaml
+++ b/test/e2e/provisioning/testdata/machinedeployment-kubevirt.yaml
@@ -38,7 +38,7 @@ spec:
                 primaryDisk:
                   osImage: http://image-repo.kube-system.svc/images/<< KUBEVIRT_OS_IMAGE >>.img
                   size: "25Gi"
-                  storageClassName: px-csi-db
+                  storageClassName: rook-ceph-block
               dnsPolicy: "None"
               dnsConfig:
                 nameservers: