Repository: kubermatic/machine-controller
Pull request #1645 (2023-05-22 13:30:10)

Allow external CCMs to handle node objects before MC
**What this PR does / why we need it**:
Allow external CCMs to handle failing node objects before MC takes any action.
It prevents race condition between MC and CCM.

The PR introduces a `handleNodeFailuresWithExternalCCM` function that:

>handleNodeFailuresWithExternalCCM reacts to node status discovery of CCM's node lifecycle controller.
>If an instance at cloud provider is not found then it waits till CCM deletes node objects, that allows:
>   - create a new instance at cloud provider
>   - initialize a new node object - the object should not be reused between instance creation
>     for example, instance foo that got deleted and recreated should initialize a completely new node object
>     instead of reusing the old one as it can cause problems to update node's metadata, like IP address.
>
> If node is shut-down it allows MC to react accordingly to specific cloud provider requirements, those are:
>   - wait for node to become online again or
>   - delete a machine which cannot be recovered

**Which issue(s) this PR fixes**:
<!--optional, in `fixes #<issue number>` format, will close the issue(s) when PR gets merged-->
Fixes https://github.com/kubermatic/kubermatic/issues/12218

**What type of PR is this?**
/kind bug
<!--
Add one of the following kinds:
/kind bug
/kind cleanup
/kind documentation
/kind feature
/kind design

Optionally add one or more of the following kinds if applicable:
/kind api-change
/kind deprecation
/kind failing-test
/kind flake
/kind regression
/kind chore
-->

**Special notes for your reviewer**:

**Does this PR introduce a user-facing change? Then add your Release Note here**:
<!--
Write your release note:
1. Enter your extended release note in the below block. If the PR requires additional action from users switching to the new release, include the string "action required".
2. If no release note is required, just write "NONE".
-->
```release-note
Allow external CCMs to handle failing node objects before MC.
```

**Documentation**:
<!--
Please do one of the following options:
- Add a link to the existing documentation
- Add a link to the kubermatic/docs pull request
- If no documentation change is applicable then add:
  - TBD (documentation will be added later)
  - NONE (no documentation needed for this PR)
-->
```documentation
NONE
```


kubermatic-bot: [APPROVALNOTIFIER] This PR is **APPROVED**

This pull-request has been approved by: *<a href="https://github.com/kubermatic/machine-controller/pull/1645#" title="Author self-approved">mfranczy</a>*

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=kubermatic%2Fmachine-controller).

The pull request process is described [here](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process)

<details >
Needs approval from an approver in each of these files:

- ~~[OWNERS](https://github.com/kubermatic/machine-controller/blob/main/OWNERS)~~ [mfranczy]
- ~~[pkg/cloudprovider/provider/kubevirt/OWNERS](https://github.com/kubermatic/machine-controller/blob/main/pkg/cloudprovider/provider/kubevirt/OWNERS)~~ [mfranczy]

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":[]} -->
mfranczy: /hold

First, I would like to merge this one https://github.com/kubevirt/cloud-provider-kubevirt/pull/243
kubermatic-bot: @mfranczy: The following test **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-machine-controller-test | 934f2cb79a8c5b81d759b3c6e940c2649066aad8 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1645/pull-machine-controller-test/1660639220978421760) | true | `/test pull-machine-controller-test`

[Full PR test history](https://public-prow.loodse.com/pr-history?org=kubermatic&repo=machine-controller&pr=1645)

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
kubermatic-bot: @mfranczy: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-machine-controller-test | 934f2cb79a8c5b81d759b3c6e940c2649066aad8 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1645/pull-machine-controller-test/1660639220978421760) | true | `/test pull-machine-controller-test`
pull-machine-controller-e2e-custom-ca | 934f2cb79a8c5b81d759b3c6e940c2649066aad8 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1645/pull-machine-controller-e2e-custom-ca/1660639218629611520) | true | `/test pull-machine-controller-e2e-custom-ca`
pull-machine-controller-build | 934f2cb79a8c5b81d759b3c6e940c2649066aad8 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1645/pull-machine-controller-build/1660639218910629888) | true | `/test pull-machine-controller-build`

[Full PR test history](https://public-prow.loodse.com/pr-history?org=kubermatic&repo=machine-controller&pr=1645)

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
kubermatic-bot: @mfranczy: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-machine-controller-test | 934f2cb79a8c5b81d759b3c6e940c2649066aad8 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1645/pull-machine-controller-test/1660639220978421760) | true | `/test pull-machine-controller-test`
pull-machine-controller-e2e-custom-ca | 934f2cb79a8c5b81d759b3c6e940c2649066aad8 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1645/pull-machine-controller-e2e-custom-ca/1660639218629611520) | true | `/test pull-machine-controller-e2e-custom-ca`
pull-machine-controller-build | 934f2cb79a8c5b81d759b3c6e940c2649066aad8 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1645/pull-machine-controller-build/1660639218910629888) | true | `/test pull-machine-controller-build`
pull-machine-controller-e2e-kubevirt | 934f2cb79a8c5b81d759b3c6e940c2649066aad8 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1645/pull-machine-controller-e2e-kubevirt/1660639218856103936) | true | `/test pull-machine-controller-e2e-kubevirt`
pull-machine-controller-e2e-aws-spot-instance | 934f2cb79a8c5b81d759b3c6e940c2649066aad8 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1645/pull-machine-controller-e2e-aws-spot-instance/1660639218797383680) | true | `/test pull-machine-controller-e2e-aws-spot-instance`

[Full PR test history](https://public-prow.loodse.com/pr-history?org=kubermatic&repo=machine-controller&pr=1645)

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
kubermatic-bot: @mfranczy: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-machine-controller-test | 934f2cb79a8c5b81d759b3c6e940c2649066aad8 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1645/pull-machine-controller-test/1660639220978421760) | true | `/test pull-machine-controller-test`
pull-machine-controller-e2e-custom-ca | 934f2cb79a8c5b81d759b3c6e940c2649066aad8 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1645/pull-machine-controller-e2e-custom-ca/1660639218629611520) | true | `/test pull-machine-controller-e2e-custom-ca`
pull-machine-controller-build | 934f2cb79a8c5b81d759b3c6e940c2649066aad8 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1645/pull-machine-controller-build/1660639218910629888) | true | `/test pull-machine-controller-build`
pull-machine-controller-e2e-kubevirt | 934f2cb79a8c5b81d759b3c6e940c2649066aad8 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1645/pull-machine-controller-e2e-kubevirt/1660639218856103936) | true | `/test pull-machine-controller-e2e-kubevirt`
pull-machine-controller-e2e-aws-spot-instance | 934f2cb79a8c5b81d759b3c6e940c2649066aad8 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1645/pull-machine-controller-e2e-aws-spot-instance/1660639218797383680) | true | `/test pull-machine-controller-e2e-aws-spot-instance`
pull-machine-controller-e2e-aws-spot-instance-legacy-userdata | 934f2cb79a8c5b81d759b3c6e940c2649066aad8 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1645/pull-machine-controller-e2e-aws-spot-instance-legacy-userdata/1660639218742857728) | true | `/test pull-machine-controller-e2e-aws-spot-instance-legacy-userdata`
pull-machine-controller-e2e-invalid-objects-get-rejected | 934f2cb79a8c5b81d759b3c6e940c2649066aad8 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1645/pull-machine-controller-e2e-invalid-objects-get-rejected/1660639218562502656) | true | `/test pull-machine-controller-e2e-invalid-objects-get-rejected`

[Full PR test history](https://public-prow.loodse.com/pr-history?org=kubermatic&repo=machine-controller&pr=1645)

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
kubermatic-bot: @mfranczy: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-machine-controller-test | 934f2cb79a8c5b81d759b3c6e940c2649066aad8 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1645/pull-machine-controller-test/1660639220978421760) | true | `/test pull-machine-controller-test`
pull-machine-controller-e2e-custom-ca | 934f2cb79a8c5b81d759b3c6e940c2649066aad8 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1645/pull-machine-controller-e2e-custom-ca/1660639218629611520) | true | `/test pull-machine-controller-e2e-custom-ca`
pull-machine-controller-build | 934f2cb79a8c5b81d759b3c6e940c2649066aad8 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1645/pull-machine-controller-build/1660639218910629888) | true | `/test pull-machine-controller-build`
pull-machine-controller-e2e-kubevirt | 934f2cb79a8c5b81d759b3c6e940c2649066aad8 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1645/pull-machine-controller-e2e-kubevirt/1660639218856103936) | true | `/test pull-machine-controller-e2e-kubevirt`
pull-machine-controller-e2e-aws-spot-instance | 934f2cb79a8c5b81d759b3c6e940c2649066aad8 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1645/pull-machine-controller-e2e-aws-spot-instance/1660639218797383680) | true | `/test pull-machine-controller-e2e-aws-spot-instance`
pull-machine-controller-e2e-aws-spot-instance-legacy-userdata | 934f2cb79a8c5b81d759b3c6e940c2649066aad8 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1645/pull-machine-controller-e2e-aws-spot-instance-legacy-userdata/1660639218742857728) | true | `/test pull-machine-controller-e2e-aws-spot-instance-legacy-userdata`
pull-machine-controller-e2e-invalid-objects-get-rejected | 934f2cb79a8c5b81d759b3c6e940c2649066aad8 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1645/pull-machine-controller-e2e-invalid-objects-get-rejected/1660639218562502656) | true | `/test pull-machine-controller-e2e-invalid-objects-get-rejected`
pull-machine-controller-e2e-ubuntu-upgrade | 934f2cb79a8c5b81d759b3c6e940c2649066aad8 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1645/pull-machine-controller-e2e-ubuntu-upgrade/1660639218684137472) | true | `/test pull-machine-controller-e2e-ubuntu-upgrade`

[Full PR test history](https://public-prow.loodse.com/pr-history?org=kubermatic&repo=machine-controller&pr=1645)

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
kubermatic-bot: @mfranczy: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-machine-controller-test | 934f2cb79a8c5b81d759b3c6e940c2649066aad8 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1645/pull-machine-controller-test/1660639220978421760) | true | `/test pull-machine-controller-test`
pull-machine-controller-e2e-custom-ca | 934f2cb79a8c5b81d759b3c6e940c2649066aad8 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1645/pull-machine-controller-e2e-custom-ca/1660639218629611520) | true | `/test pull-machine-controller-e2e-custom-ca`
pull-machine-controller-build | 934f2cb79a8c5b81d759b3c6e940c2649066aad8 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1645/pull-machine-controller-build/1660639218910629888) | true | `/test pull-machine-controller-build`
pull-machine-controller-e2e-kubevirt | 934f2cb79a8c5b81d759b3c6e940c2649066aad8 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1645/pull-machine-controller-e2e-kubevirt/1660639218856103936) | true | `/test pull-machine-controller-e2e-kubevirt`
pull-machine-controller-e2e-aws-spot-instance | 934f2cb79a8c5b81d759b3c6e940c2649066aad8 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1645/pull-machine-controller-e2e-aws-spot-instance/1660639218797383680) | true | `/test pull-machine-controller-e2e-aws-spot-instance`
pull-machine-controller-e2e-aws-spot-instance-legacy-userdata | 934f2cb79a8c5b81d759b3c6e940c2649066aad8 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1645/pull-machine-controller-e2e-aws-spot-instance-legacy-userdata/1660639218742857728) | true | `/test pull-machine-controller-e2e-aws-spot-instance-legacy-userdata`
pull-machine-controller-e2e-invalid-objects-get-rejected | 934f2cb79a8c5b81d759b3c6e940c2649066aad8 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1645/pull-machine-controller-e2e-invalid-objects-get-rejected/1660639218562502656) | true | `/test pull-machine-controller-e2e-invalid-objects-get-rejected`
pull-machine-controller-e2e-ubuntu-upgrade | 934f2cb79a8c5b81d759b3c6e940c2649066aad8 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1645/pull-machine-controller-e2e-ubuntu-upgrade/1660639218684137472) | true | `/test pull-machine-controller-e2e-ubuntu-upgrade`
pull-machine-controller-lint | 934f2cb79a8c5b81d759b3c6e940c2649066aad8 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1645/pull-machine-controller-lint/1660639219007098880) | true | `/test pull-machine-controller-lint`

[Full PR test history](https://public-prow.loodse.com/pr-history?org=kubermatic&repo=machine-controller&pr=1645)

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
kubermatic-bot: @mfranczy: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-machine-controller-e2e-custom-ca | 934f2cb79a8c5b81d759b3c6e940c2649066aad8 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1645/pull-machine-controller-e2e-custom-ca/1660639218629611520) | true | `/test pull-machine-controller-e2e-custom-ca`
pull-machine-controller-build | 934f2cb79a8c5b81d759b3c6e940c2649066aad8 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1645/pull-machine-controller-build/1660639218910629888) | true | `/test pull-machine-controller-build`
pull-machine-controller-e2e-kubevirt | 934f2cb79a8c5b81d759b3c6e940c2649066aad8 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1645/pull-machine-controller-e2e-kubevirt/1660639218856103936) | true | `/test pull-machine-controller-e2e-kubevirt`
pull-machine-controller-e2e-aws-spot-instance | 934f2cb79a8c5b81d759b3c6e940c2649066aad8 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1645/pull-machine-controller-e2e-aws-spot-instance/1660639218797383680) | true | `/test pull-machine-controller-e2e-aws-spot-instance`
pull-machine-controller-e2e-aws-spot-instance-legacy-userdata | 934f2cb79a8c5b81d759b3c6e940c2649066aad8 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1645/pull-machine-controller-e2e-aws-spot-instance-legacy-userdata/1660639218742857728) | true | `/test pull-machine-controller-e2e-aws-spot-instance-legacy-userdata`
pull-machine-controller-e2e-invalid-objects-get-rejected | 934f2cb79a8c5b81d759b3c6e940c2649066aad8 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1645/pull-machine-controller-e2e-invalid-objects-get-rejected/1660639218562502656) | true | `/test pull-machine-controller-e2e-invalid-objects-get-rejected`
pull-machine-controller-e2e-ubuntu-upgrade | 934f2cb79a8c5b81d759b3c6e940c2649066aad8 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1645/pull-machine-controller-e2e-ubuntu-upgrade/1660639218684137472) | true | `/test pull-machine-controller-e2e-ubuntu-upgrade`
pull-machine-controller-lint | 934f2cb79a8c5b81d759b3c6e940c2649066aad8 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1645/pull-machine-controller-lint/1660639219007098880) | true | `/test pull-machine-controller-lint`
pull-machine-controller-test | e6ebf16354205312ca746209b59be202a8517894 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1645/pull-machine-controller-test/1660991256806821888) | true | `/test pull-machine-controller-test`

[Full PR test history](https://public-prow.loodse.com/pr-history?org=kubermatic&repo=machine-controller&pr=1645)

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
kubermatic-bot: @mfranczy: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-machine-controller-e2e-custom-ca | 934f2cb79a8c5b81d759b3c6e940c2649066aad8 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1645/pull-machine-controller-e2e-custom-ca/1660639218629611520) | true | `/test pull-machine-controller-e2e-custom-ca`
pull-machine-controller-build | 934f2cb79a8c5b81d759b3c6e940c2649066aad8 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1645/pull-machine-controller-build/1660639218910629888) | true | `/test pull-machine-controller-build`
pull-machine-controller-e2e-kubevirt | 934f2cb79a8c5b81d759b3c6e940c2649066aad8 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1645/pull-machine-controller-e2e-kubevirt/1660639218856103936) | true | `/test pull-machine-controller-e2e-kubevirt`
pull-machine-controller-e2e-aws-spot-instance | 934f2cb79a8c5b81d759b3c6e940c2649066aad8 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1645/pull-machine-controller-e2e-aws-spot-instance/1660639218797383680) | true | `/test pull-machine-controller-e2e-aws-spot-instance`
pull-machine-controller-e2e-aws-spot-instance-legacy-userdata | 934f2cb79a8c5b81d759b3c6e940c2649066aad8 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1645/pull-machine-controller-e2e-aws-spot-instance-legacy-userdata/1660639218742857728) | true | `/test pull-machine-controller-e2e-aws-spot-instance-legacy-userdata`
pull-machine-controller-e2e-invalid-objects-get-rejected | 934f2cb79a8c5b81d759b3c6e940c2649066aad8 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1645/pull-machine-controller-e2e-invalid-objects-get-rejected/1660639218562502656) | true | `/test pull-machine-controller-e2e-invalid-objects-get-rejected`
pull-machine-controller-e2e-ubuntu-upgrade | 934f2cb79a8c5b81d759b3c6e940c2649066aad8 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1645/pull-machine-controller-e2e-ubuntu-upgrade/1660639218684137472) | true | `/test pull-machine-controller-e2e-ubuntu-upgrade`
pull-machine-controller-lint | 934f2cb79a8c5b81d759b3c6e940c2649066aad8 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1645/pull-machine-controller-lint/1660639219007098880) | true | `/test pull-machine-controller-lint`
pull-machine-controller-test | e6ebf16354205312ca746209b59be202a8517894 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1645/pull-machine-controller-test/1660991256806821888) | true | `/test pull-machine-controller-test`
pull-machine-controller-dependencies | e6ebf16354205312ca746209b59be202a8517894 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1645/pull-machine-controller-dependencies/1660991254084718592) | true | `/test pull-machine-controller-dependencies`

[Full PR test history](https://public-prow.loodse.com/pr-history?org=kubermatic&repo=machine-controller&pr=1645)

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
kubermatic-bot: @mfranczy: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-machine-controller-e2e-custom-ca | 934f2cb79a8c5b81d759b3c6e940c2649066aad8 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1645/pull-machine-controller-e2e-custom-ca/1660639218629611520) | true | `/test pull-machine-controller-e2e-custom-ca`
pull-machine-controller-e2e-kubevirt | 934f2cb79a8c5b81d759b3c6e940c2649066aad8 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1645/pull-machine-controller-e2e-kubevirt/1660639218856103936) | true | `/test pull-machine-controller-e2e-kubevirt`
pull-machine-controller-e2e-aws-spot-instance | 934f2cb79a8c5b81d759b3c6e940c2649066aad8 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1645/pull-machine-controller-e2e-aws-spot-instance/1660639218797383680) | true | `/test pull-machine-controller-e2e-aws-spot-instance`
pull-machine-controller-e2e-aws-spot-instance-legacy-userdata | 934f2cb79a8c5b81d759b3c6e940c2649066aad8 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1645/pull-machine-controller-e2e-aws-spot-instance-legacy-userdata/1660639218742857728) | true | `/test pull-machine-controller-e2e-aws-spot-instance-legacy-userdata`
pull-machine-controller-e2e-invalid-objects-get-rejected | 934f2cb79a8c5b81d759b3c6e940c2649066aad8 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1645/pull-machine-controller-e2e-invalid-objects-get-rejected/1660639218562502656) | true | `/test pull-machine-controller-e2e-invalid-objects-get-rejected`
pull-machine-controller-e2e-ubuntu-upgrade | 934f2cb79a8c5b81d759b3c6e940c2649066aad8 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1645/pull-machine-controller-e2e-ubuntu-upgrade/1660639218684137472) | true | `/test pull-machine-controller-e2e-ubuntu-upgrade`
pull-machine-controller-lint | 934f2cb79a8c5b81d759b3c6e940c2649066aad8 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1645/pull-machine-controller-lint/1660639219007098880) | true | `/test pull-machine-controller-lint`
pull-machine-controller-test | e6ebf16354205312ca746209b59be202a8517894 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1645/pull-machine-controller-test/1660991256806821888) | true | `/test pull-machine-controller-test`
pull-machine-controller-dependencies | e6ebf16354205312ca746209b59be202a8517894 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1645/pull-machine-controller-dependencies/1660991254084718592) | true | `/test pull-machine-controller-dependencies`
pull-machine-controller-build | e6ebf16354205312ca746209b59be202a8517894 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1645/pull-machine-controller-build/1660991254034386944) | true | `/test pull-machine-controller-build`

[Full PR test history](https://public-prow.loodse.com/pr-history?org=kubermatic&repo=machine-controller&pr=1645)

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
kubermatic-bot: @mfranczy: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-machine-controller-e2e-custom-ca | 934f2cb79a8c5b81d759b3c6e940c2649066aad8 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1645/pull-machine-controller-e2e-custom-ca/1660639218629611520) | true | `/test pull-machine-controller-e2e-custom-ca`
pull-machine-controller-e2e-kubevirt | 934f2cb79a8c5b81d759b3c6e940c2649066aad8 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1645/pull-machine-controller-e2e-kubevirt/1660639218856103936) | true | `/test pull-machine-controller-e2e-kubevirt`
pull-machine-controller-e2e-aws-spot-instance | 934f2cb79a8c5b81d759b3c6e940c2649066aad8 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1645/pull-machine-controller-e2e-aws-spot-instance/1660639218797383680) | true | `/test pull-machine-controller-e2e-aws-spot-instance`
pull-machine-controller-e2e-aws-spot-instance-legacy-userdata | 934f2cb79a8c5b81d759b3c6e940c2649066aad8 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1645/pull-machine-controller-e2e-aws-spot-instance-legacy-userdata/1660639218742857728) | true | `/test pull-machine-controller-e2e-aws-spot-instance-legacy-userdata`
pull-machine-controller-lint | 934f2cb79a8c5b81d759b3c6e940c2649066aad8 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1645/pull-machine-controller-lint/1660639219007098880) | true | `/test pull-machine-controller-lint`
pull-machine-controller-test | e6ebf16354205312ca746209b59be202a8517894 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1645/pull-machine-controller-test/1660991256806821888) | true | `/test pull-machine-controller-test`
pull-machine-controller-dependencies | e6ebf16354205312ca746209b59be202a8517894 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1645/pull-machine-controller-dependencies/1660991254084718592) | true | `/test pull-machine-controller-dependencies`
pull-machine-controller-build | e6ebf16354205312ca746209b59be202a8517894 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1645/pull-machine-controller-build/1660991254034386944) | true | `/test pull-machine-controller-build`
pull-machine-controller-e2e-invalid-objects-get-rejected | e6ebf16354205312ca746209b59be202a8517894 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1645/pull-machine-controller-e2e-invalid-objects-get-rejected/1660991253631733760) | true | `/test pull-machine-controller-e2e-invalid-objects-get-rejected`
pull-machine-controller-e2e-ubuntu-upgrade | e6ebf16354205312ca746209b59be202a8517894 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1645/pull-machine-controller-e2e-ubuntu-upgrade/1660991253770145792) | true | `/test pull-machine-controller-e2e-ubuntu-upgrade`

[Full PR test history](https://public-prow.loodse.com/pr-history?org=kubermatic&repo=machine-controller&pr=1645)

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
kubermatic-bot: @mfranczy: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-machine-controller-e2e-custom-ca | 934f2cb79a8c5b81d759b3c6e940c2649066aad8 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1645/pull-machine-controller-e2e-custom-ca/1660639218629611520) | true | `/test pull-machine-controller-e2e-custom-ca`
pull-machine-controller-e2e-kubevirt | 934f2cb79a8c5b81d759b3c6e940c2649066aad8 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1645/pull-machine-controller-e2e-kubevirt/1660639218856103936) | true | `/test pull-machine-controller-e2e-kubevirt`
pull-machine-controller-e2e-aws-spot-instance-legacy-userdata | 934f2cb79a8c5b81d759b3c6e940c2649066aad8 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1645/pull-machine-controller-e2e-aws-spot-instance-legacy-userdata/1660639218742857728) | true | `/test pull-machine-controller-e2e-aws-spot-instance-legacy-userdata`
pull-machine-controller-lint | 934f2cb79a8c5b81d759b3c6e940c2649066aad8 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1645/pull-machine-controller-lint/1660639219007098880) | true | `/test pull-machine-controller-lint`
pull-machine-controller-test | e6ebf16354205312ca746209b59be202a8517894 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1645/pull-machine-controller-test/1660991256806821888) | true | `/test pull-machine-controller-test`
pull-machine-controller-dependencies | e6ebf16354205312ca746209b59be202a8517894 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1645/pull-machine-controller-dependencies/1660991254084718592) | true | `/test pull-machine-controller-dependencies`
pull-machine-controller-build | e6ebf16354205312ca746209b59be202a8517894 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1645/pull-machine-controller-build/1660991254034386944) | true | `/test pull-machine-controller-build`
pull-machine-controller-e2e-invalid-objects-get-rejected | e6ebf16354205312ca746209b59be202a8517894 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1645/pull-machine-controller-e2e-invalid-objects-get-rejected/1660991253631733760) | true | `/test pull-machine-controller-e2e-invalid-objects-get-rejected`
pull-machine-controller-e2e-ubuntu-upgrade | e6ebf16354205312ca746209b59be202a8517894 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1645/pull-machine-controller-e2e-ubuntu-upgrade/1660991253770145792) | true | `/test pull-machine-controller-e2e-ubuntu-upgrade`
pull-machine-controller-e2e-aws-spot-instance | e6ebf16354205312ca746209b59be202a8517894 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1645/pull-machine-controller-e2e-aws-spot-instance/1660991253904363520) | true | `/test pull-machine-controller-e2e-aws-spot-instance`

[Full PR test history](https://public-prow.loodse.com/pr-history?org=kubermatic&repo=machine-controller&pr=1645)

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
mfranczy: /retest
kubermatic-bot: @mfranczy: The following test **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-machine-controller-e2e-kubevirt | 67bf30beb57bf67c141f1a40c0a4e99ea1d57b87 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1645/pull-machine-controller-e2e-kubevirt/1660993177944854528) | true | `/test pull-machine-controller-e2e-kubevirt`

[Full PR test history](https://public-prow.loodse.com/pr-history?org=kubermatic&repo=machine-controller&pr=1645)

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
kubermatic-bot: @mfranczy: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-machine-controller-e2e-kubevirt | 67bf30beb57bf67c141f1a40c0a4e99ea1d57b87 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1645/pull-machine-controller-e2e-kubevirt/1660993177944854528) | true | `/test pull-machine-controller-e2e-kubevirt`
pull-machine-controller-e2e-aws-spot-instance-legacy-userdata | 67bf30beb57bf67c141f1a40c0a4e99ea1d57b87 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1645/pull-machine-controller-e2e-aws-spot-instance-legacy-userdata/1660993177743527936) | true | `/test pull-machine-controller-e2e-aws-spot-instance-legacy-userdata`

[Full PR test history](https://public-prow.loodse.com/pr-history?org=kubermatic&repo=machine-controller&pr=1645)

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
kubermatic-bot: @mfranczy: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-machine-controller-e2e-kubevirt | 67bf30beb57bf67c141f1a40c0a4e99ea1d57b87 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1645/pull-machine-controller-e2e-kubevirt/1660993177944854528) | true | `/test pull-machine-controller-e2e-kubevirt`
pull-machine-controller-e2e-aws-spot-instance-legacy-userdata | 67bf30beb57bf67c141f1a40c0a4e99ea1d57b87 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1645/pull-machine-controller-e2e-aws-spot-instance-legacy-userdata/1660993177743527936) | true | `/test pull-machine-controller-e2e-aws-spot-instance-legacy-userdata`
pull-machine-controller-e2e-aws-spot-instance | 67bf30beb57bf67c141f1a40c0a4e99ea1d57b87 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1645/pull-machine-controller-e2e-aws-spot-instance/1660993177806442496) | true | `/test pull-machine-controller-e2e-aws-spot-instance`

[Full PR test history](https://public-prow.loodse.com/pr-history?org=kubermatic&repo=machine-controller&pr=1645)

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
mfranczy: How irrational... KubeVirt tests fail. Investigating...
mfranczy: /retest
kubermatic-bot: @mfranczy: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-machine-controller-e2e-aws-spot-instance-legacy-userdata | 67bf30beb57bf67c141f1a40c0a4e99ea1d57b87 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1645/pull-machine-controller-e2e-aws-spot-instance-legacy-userdata/1660993177743527936) | true | `/test pull-machine-controller-e2e-aws-spot-instance-legacy-userdata`
pull-machine-controller-e2e-aws-spot-instance | 67bf30beb57bf67c141f1a40c0a4e99ea1d57b87 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1645/pull-machine-controller-e2e-aws-spot-instance/1660993177806442496) | true | `/test pull-machine-controller-e2e-aws-spot-instance`
pull-machine-controller-e2e-kubevirt | 67bf30beb57bf67c141f1a40c0a4e99ea1d57b87 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1645/pull-machine-controller-e2e-kubevirt/1663057480030097408) | true | `/test pull-machine-controller-e2e-kubevirt`

[Full PR test history](https://public-prow.loodse.com/pr-history?org=kubermatic&repo=machine-controller&pr=1645)

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
kubermatic-bot: @mfranczy: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-machine-controller-e2e-aws-spot-instance-legacy-userdata | 67bf30beb57bf67c141f1a40c0a4e99ea1d57b87 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1645/pull-machine-controller-e2e-aws-spot-instance-legacy-userdata/1660993177743527936) | true | `/test pull-machine-controller-e2e-aws-spot-instance-legacy-userdata`
pull-machine-controller-e2e-kubevirt | 67bf30beb57bf67c141f1a40c0a4e99ea1d57b87 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1645/pull-machine-controller-e2e-kubevirt/1663057480030097408) | true | `/test pull-machine-controller-e2e-kubevirt`
pull-machine-controller-e2e-aws-spot-instance | 67bf30beb57bf67c141f1a40c0a4e99ea1d57b87 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1645/pull-machine-controller-e2e-aws-spot-instance/1663057479979765760) | true | `/test pull-machine-controller-e2e-aws-spot-instance`

[Full PR test history](https://public-prow.loodse.com/pr-history?org=kubermatic&repo=machine-controller&pr=1645)

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
kubermatic-bot: @mfranczy: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-machine-controller-e2e-kubevirt | 67bf30beb57bf67c141f1a40c0a4e99ea1d57b87 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1645/pull-machine-controller-e2e-kubevirt/1663057480030097408) | true | `/test pull-machine-controller-e2e-kubevirt`
pull-machine-controller-e2e-aws-spot-instance | 67bf30beb57bf67c141f1a40c0a4e99ea1d57b87 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1645/pull-machine-controller-e2e-aws-spot-instance/1663057479979765760) | true | `/test pull-machine-controller-e2e-aws-spot-instance`
pull-machine-controller-e2e-aws-spot-instance-legacy-userdata | 67bf30beb57bf67c141f1a40c0a4e99ea1d57b87 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1645/pull-machine-controller-e2e-aws-spot-instance-legacy-userdata/1663057479929434112) | true | `/test pull-machine-controller-e2e-aws-spot-instance-legacy-userdata`

[Full PR test history](https://public-prow.loodse.com/pr-history?org=kubermatic&repo=machine-controller&pr=1645)

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
mfranczy: /test  pull-machine-controller-e2e-kubevirt
kubermatic-bot: @mfranczy: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-machine-controller-e2e-aws-spot-instance | 67bf30beb57bf67c141f1a40c0a4e99ea1d57b87 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1645/pull-machine-controller-e2e-aws-spot-instance/1663057479979765760) | true | `/test pull-machine-controller-e2e-aws-spot-instance`
pull-machine-controller-e2e-aws-spot-instance-legacy-userdata | 67bf30beb57bf67c141f1a40c0a4e99ea1d57b87 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1645/pull-machine-controller-e2e-aws-spot-instance-legacy-userdata/1663057479929434112) | true | `/test pull-machine-controller-e2e-aws-spot-instance-legacy-userdata`
pull-machine-controller-e2e-kubevirt | 67bf30beb57bf67c141f1a40c0a4e99ea1d57b87 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1645/pull-machine-controller-e2e-kubevirt/1663075189547601920) | true | `/test pull-machine-controller-e2e-kubevirt`

[Full PR test history](https://public-prow.loodse.com/pr-history?org=kubermatic&repo=machine-controller&pr=1645)

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
mfranczy: /test pull-machine-controller-e2e-aws
kubermatic-bot: @mfranczy: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-machine-controller-e2e-aws-spot-instance | 67bf30beb57bf67c141f1a40c0a4e99ea1d57b87 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1645/pull-machine-controller-e2e-aws-spot-instance/1663057479979765760) | true | `/test pull-machine-controller-e2e-aws-spot-instance`
pull-machine-controller-e2e-aws-spot-instance-legacy-userdata | 67bf30beb57bf67c141f1a40c0a4e99ea1d57b87 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1645/pull-machine-controller-e2e-aws-spot-instance-legacy-userdata/1663057479929434112) | true | `/test pull-machine-controller-e2e-aws-spot-instance-legacy-userdata`
pull-machine-controller-e2e-kubevirt | 67bf30beb57bf67c141f1a40c0a4e99ea1d57b87 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1645/pull-machine-controller-e2e-kubevirt/1663075189547601920) | true | `/test pull-machine-controller-e2e-kubevirt`
pull-machine-controller-e2e-gce | 67bf30beb57bf67c141f1a40c0a4e99ea1d57b87 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1645/pull-machine-controller-e2e-gce/1663078864118616064) | true | `/test pull-machine-controller-e2e-gce`

[Full PR test history](https://public-prow.loodse.com/pr-history?org=kubermatic&repo=machine-controller&pr=1645)

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
kubermatic-bot: @mfranczy: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-machine-controller-e2e-aws-spot-instance | 67bf30beb57bf67c141f1a40c0a4e99ea1d57b87 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1645/pull-machine-controller-e2e-aws-spot-instance/1663057479979765760) | true | `/test pull-machine-controller-e2e-aws-spot-instance`
pull-machine-controller-e2e-aws-spot-instance-legacy-userdata | 67bf30beb57bf67c141f1a40c0a4e99ea1d57b87 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1645/pull-machine-controller-e2e-aws-spot-instance-legacy-userdata/1663057479929434112) | true | `/test pull-machine-controller-e2e-aws-spot-instance-legacy-userdata`
pull-machine-controller-e2e-kubevirt | 67bf30beb57bf67c141f1a40c0a4e99ea1d57b87 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1645/pull-machine-controller-e2e-kubevirt/1663075189547601920) | true | `/test pull-machine-controller-e2e-kubevirt`
pull-machine-controller-e2e-gce | 67bf30beb57bf67c141f1a40c0a4e99ea1d57b87 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1645/pull-machine-controller-e2e-gce/1663078864118616064) | true | `/test pull-machine-controller-e2e-gce`
pull-machine-controller-e2e-aws | 67bf30beb57bf67c141f1a40c0a4e99ea1d57b87 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1645/pull-machine-controller-e2e-aws/1663078953557954560) | true | `/test pull-machine-controller-e2e-aws`

[Full PR test history](https://public-prow.loodse.com/pr-history?org=kubermatic&repo=machine-controller&pr=1645)

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
mfranczy: /retest
mfranczy: KubeVirt tests fails becasue of problem with storage class on the infra cluster. Investigating...
kubermatic-bot: @mfranczy: The following test **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-machine-controller-e2e-kubevirt | 67bf30beb57bf67c141f1a40c0a4e99ea1d57b87 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1645/pull-machine-controller-e2e-kubevirt/1663085091879915520) | true | `/test pull-machine-controller-e2e-kubevirt`

[Full PR test history](https://public-prow.loodse.com/pr-history?org=kubermatic&repo=machine-controller&pr=1645)

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
mfranczy: /test pull-machine-controller-e2e-gce
mfranczy: @kron4eg @embik tests are ok now. I consider this PR to be ready to review.

Patch:
diff --git a/go.mod b/go.mod
index 8418da9a..551e32cf 100644
--- a/go.mod
+++ b/go.mod
@@ -58,6 +58,7 @@ require (
 	k8s.io/apiextensions-apiserver v0.26.4
 	k8s.io/apimachinery v0.26.4
 	k8s.io/client-go v12.0.0+incompatible
+	k8s.io/cloud-provider v0.26.4
 	k8s.io/klog v1.0.0
 	k8s.io/kubelet v0.26.4
 	k8s.io/utils v0.0.0-20230209194617-a36077c30491
diff --git a/go.sum b/go.sum
index ab7ec118..eeb0a4ab 100644
--- a/go.sum
+++ b/go.sum
@@ -1235,6 +1235,8 @@ k8s.io/apimachinery v0.26.4 h1:rZccKdBLg9vP6J09JD+z8Yr99Ce8gk3Lbi9TCx05Jzs=
 k8s.io/apimachinery v0.26.4/go.mod h1:ats7nN1LExKHvJ9TmwootT00Yz05MuYqPXEXaVeOy5I=
 k8s.io/client-go v0.26.4 h1:/7P/IbGBuT73A+G97trf44NTPSNqvuBREpOfdLbHvD4=
 k8s.io/client-go v0.26.4/go.mod h1:6qOItWm3EwxJdl/8p5t7FWtWUOwyMdA8N9ekbW4idpI=
+k8s.io/cloud-provider v0.26.4 h1:mqN4vhC4mRoMi+ujI92ImkIOuYS7ZS55FvXB10d6Wp4=
+k8s.io/cloud-provider v0.26.4/go.mod h1:F9xY0PvBuZDuGIHOM28dNiPLHxQnWfsiUuCSUikHevo=
 k8s.io/code-generator v0.23.0/go.mod h1:vQvOhDXhuzqiVfM/YHp+dmg10WDZCchJVObc9MvowsE=
 k8s.io/code-generator v0.23.3/go.mod h1:S0Q1JVA+kSzTI1oUvbKAxZY/DYbA/ZUb4Uknog12ETk=
 k8s.io/component-base v0.26.4 h1:Bg2xzyXNKL3eAuiTEu3XE198d6z22ENgFgGQv2GGOUk=
diff --git a/pkg/cloudprovider/provider/kubevirt/provider.go b/pkg/cloudprovider/provider/kubevirt/provider.go
index e5aba104..a2ba1f58 100644
--- a/pkg/cloudprovider/provider/kubevirt/provider.go
+++ b/pkg/cloudprovider/provider/kubevirt/provider.go
@@ -445,18 +445,6 @@ func (p *provider) Get(ctx context.Context, _ *zap.SugaredLogger, machine *clust
 		return nil, cloudprovidererrors.ErrInstanceNotFound
 	}
 
-	if virtualMachineInstance.Status.Phase == kubevirtv1.Failed ||
-		// The VMI enters phase succeeded if someone issues a kubectl
-		// delete pod on the virt-launcher pod it runs in
-		virtualMachineInstance.Status.Phase == kubevirtv1.Succeeded {
-		// The pod got deleted, delete the VMI and return ErrNotFound so the VMI
-		// will get recreated
-		if err := sigClient.Delete(ctx, virtualMachineInstance); err != nil {
-			return nil, fmt.Errorf("failed to delete failed VMI %s: %w", machine.Name, err)
-		}
-		return nil, cloudprovidererrors.ErrInstanceNotFound
-	}
-
 	return &kubeVirtServer{vmi: *virtualMachineInstance}, nil
 }
 
@@ -633,6 +621,8 @@ func (p *provider) newVirtualMachine(ctx context.Context, c *Config, pc *provide
 		return nil, fmt.Errorf("could not compute a random MAC address")
 	}
 
+	runStrategyOnce := kubevirtv1.RunStrategyOnce
+
 	virtualMachine := &kubevirtv1.VirtualMachine{
 		ObjectMeta: metav1.ObjectMeta{
 			Name:      machine.Name,
@@ -640,7 +630,7 @@ func (p *provider) newVirtualMachine(ctx context.Context, c *Config, pc *provide
 			Labels:    labels,
 		},
 		Spec: kubevirtv1.VirtualMachineSpec{
-			Running:      utilpointer.Bool(true),
+			RunStrategy:  &runStrategyOnce,
 			Instancetype: c.Instancetype,
 			Preference:   c.Preference,
 			Template: &kubevirtv1.VirtualMachineInstanceTemplateSpec{
@@ -692,7 +682,6 @@ func (p *provider) Cleanup(ctx context.Context, _ *zap.SugaredLogger, machine *c
 		if !kerrors.IsNotFound(err) {
 			return false, fmt.Errorf("failed to get VirtualMachineInstance %s: %w", machine.Name, err)
 		}
-		// VMI is gone
 		return true, nil
 	}
 
diff --git a/pkg/controller/machine/controller.go b/pkg/controller/machine/controller.go
index 67a53749..d952ec11 100644
--- a/pkg/controller/machine/controller.go
+++ b/pkg/controller/machine/controller.go
@@ -67,6 +67,8 @@ import (
 	"k8s.io/client-go/tools/record"
 	"k8s.io/client-go/tools/reference"
 	"k8s.io/client-go/util/retry"
+
+	ccmapi "k8s.io/cloud-provider/api"
 	"sigs.k8s.io/controller-runtime/pkg/client"
 	ctrlruntimeclient "sigs.k8s.io/controller-runtime/pkg/client"
 	"sigs.k8s.io/controller-runtime/pkg/controller"
@@ -434,9 +436,8 @@ func (r *Reconciler) reconcile(ctx context.Context, log *zap.SugaredLogger, mach
 
 	// step 2: check if a user requested to delete the machine
 	if machine.DeletionTimestamp != nil {
-		return r.deleteMachine(ctx, log, prov, providerConfig.CloudProvider, machine)
-		// deleteResult, deleteErr := r.deleteMachine(ctx, log, prov, providerConfig.CloudProvider, machine)
-		// if client.IgnoreNotFound(deleteErr)
+		skipEviction := false
+		return r.deleteMachine(ctx, log, prov, providerConfig.CloudProvider, machine, skipEviction)
 	}
 
 	// Step 3: Essentially creates an instance for the given machine.
@@ -471,8 +472,10 @@ func (r *Reconciler) reconcile(ctx context.Context, log *zap.SugaredLogger, mach
 			return nil, fmt.Errorf("failed to set nodeReady condition on machine: %w", err)
 		}
 	} else {
-		// Node is not ready anymore? Maybe it got deleted
-		return r.ensureInstanceExistsForMachine(ctx, nodeLog, prov, machine, userdataPlugin, providerConfig)
+		if r.nodeSettings.ExternalCloudProvider {
+			return r.handleNodeFailuresWithExternalCCM(ctx, log, prov, providerConfig, node, machine)
+		}
+		return r.ensureInstanceExistsForMachine(ctx, log, prov, machine, userdataPlugin, providerConfig)
 	}
 
 	// case 3.3: if the node exists make sure if it has labels and taints attached to it.
@@ -534,7 +537,7 @@ func (r *Reconciler) shouldCleanupVolumes(ctx context.Context, log *zap.SugaredL
 func (r *Reconciler) shouldEvict(ctx context.Context, log *zap.SugaredLogger, machine *clusterv1alpha1.Machine) (bool, error) {
 	// If the deletion got triggered a few hours ago, skip eviction.
 	// We assume here that the eviction is blocked by misconfiguration or a misbehaving kubelet and/or controller-runtime
-	if time.Since(machine.DeletionTimestamp.Time) > r.skipEvictionAfter {
+	if machine.DeletionTimestamp != nil && time.Since(machine.DeletionTimestamp.Time) > r.skipEvictionAfter {
 		log.Infow("Skipping eviction since the deletion got triggered too long ago", "threshold", r.skipEvictionAfter)
 		return false, nil
 	}
@@ -583,10 +586,24 @@ func (r *Reconciler) shouldEvict(ctx context.Context, log *zap.SugaredLogger, ma
 }
 
 // deleteMachine makes sure that an instance has gone in a series of steps.
-func (r *Reconciler) deleteMachine(ctx context.Context, log *zap.SugaredLogger, prov cloudprovidertypes.Provider, providerName providerconfigtypes.CloudProvider, machine *clusterv1alpha1.Machine) (*reconcile.Result, error) {
-	shouldEvict, err := r.shouldEvict(ctx, log, machine)
-	if err != nil {
-		return nil, err
+func (r *Reconciler) deleteMachine(
+	ctx context.Context,
+	log *zap.SugaredLogger,
+	prov cloudprovidertypes.Provider,
+	providerName providerconfigtypes.CloudProvider,
+	machine *clusterv1alpha1.Machine,
+	skipEviction bool,
+) (*reconcile.Result, error) {
+	var (
+		shouldEvict bool
+		err         error
+	)
+
+	if !skipEviction {
+		shouldEvict, err = r.shouldEvict(ctx, log, machine)
+		if err != nil {
+			return nil, err
+		}
 	}
 	shouldCleanUpVolumes, err := r.shouldCleanupVolumes(ctx, log, machine, providerName)
 	if err != nil {
@@ -691,7 +708,6 @@ func (r *Reconciler) deleteCloudProviderInstance(ctx context.Context, log *zap.S
 		message := fmt.Sprintf("%v. Please manually delete %s finalizer from the machine object.", err, FinalizerDeleteInstance)
 		return nil, r.updateMachineErrorIfTerminalError(machine, common.DeleteMachineError, message, err, "failed to delete machine at cloud provider")
 	}
-
 	if !completelyGone {
 		// As the instance is not completely gone yet, we need to recheck in a few seconds.
 		return &reconcile.Result{RequeueAfter: deletionRetryWaitPeriod}, nil
@@ -1051,14 +1067,6 @@ func (r *Reconciler) ensureNodeLabelsAnnotationsAndTaints(ctx context.Context, n
 		modifiers = append(modifiers, f(AnnotationAutoscalerIdentifier, autoscalerAnnotationValue))
 	}
 
-	taintExists := func(node *corev1.Node, taint corev1.Taint) bool {
-		for _, t := range node.Spec.Taints {
-			if t.MatchTaint(&taint) {
-				return true
-			}
-		}
-		return false
-	}
 	for _, t := range machine.Spec.Taints {
 		if !taintExists(node, t) {
 			f := func(t corev1.Taint) func(*corev1.Node) {
@@ -1181,6 +1189,15 @@ func findNodeByProviderID(instance instance.Instance, provider providerconfigtyp
 	return nil
 }
 
+func taintExists(node *corev1.Node, taint corev1.Taint) bool {
+	for _, t := range node.Spec.Taints {
+		if t.MatchTaint(&taint) {
+			return true
+		}
+	}
+	return false
+}
+
 func (r *Reconciler) ReadinessChecks(ctx context.Context) map[string]healthcheck.Check {
 	return map[string]healthcheck.Check{
 		"valid-info-kubeconfig": func() error {
@@ -1242,3 +1259,46 @@ func (r *Reconciler) updateNode(ctx context.Context, node *corev1.Node, modifier
 		return r.client.Update(ctx, node)
 	})
 }
+
+// handleNodeFailuresWithExternalCCM reacts to node status discovery of CCM's node lifecycle controller.
+// If an instance at cloud provider is not found then it waits till CCM deletes node objects, that allows:
+//   - create a new instance at cloud provider
+//   - initialize a new node object - the object should not be reused between instance creation
+//     for example, instance foo that got deleted and recreated should initialize a completely new node object
+//     instead of reusing the old one as it can cause problems to update node's metadata, like IP address.
+//
+// If node is shut-down it allows MC to react accordingly to specific cloud provider requirements, those are:
+//   - wait for node to become online again or
+//   - delete a machine which cannot be recovered
+func (r *Reconciler) handleNodeFailuresWithExternalCCM(
+	ctx context.Context,
+	log *zap.SugaredLogger,
+	prov cloudprovidertypes.Provider,
+	provConfig *providerconfigtypes.Config,
+	node *corev1.Node,
+	machine *clusterv1alpha1.Machine,
+) (*reconcile.Result, error) {
+	taintShutdown := corev1.Taint{
+		Key:    ccmapi.TaintNodeShutdown,
+		Effect: corev1.TaintEffectNoSchedule,
+	}
+
+	_, err := prov.Get(ctx, log, machine, r.providerData)
+	if err != nil {
+		if cloudprovidererrors.IsNotFound(err) {
+			log.Info("The node does not have corresponding instance, waiting for CCM to delete it")
+			return &reconcile.Result{RequeueAfter: deletionRetryWaitPeriod}, nil
+		}
+		return nil, err
+	} else if taintExists(node, taintShutdown) {
+		switch provConfig.CloudProvider {
+		case providerconfigtypes.CloudProviderKubeVirt:
+			log.Infof("Deleting a shut-down machine %q that cannot recover", machine.Name)
+			skipEviction := true
+			return r.deleteMachine(ctx, log, prov, providerconfigtypes.CloudProviderKubeVirt, machine, skipEviction)
+		}
+	}
+
+	log.Debug("Waiting for a node to become %q", corev1.NodeReady)
+	return &reconcile.Result{RequeueAfter: deletionRetryWaitPeriod}, err
+}


Test patch:
diff --git a/pkg/cloudprovider/provider/kubevirt/testdata/affinity-no-values.yaml b/pkg/cloudprovider/provider/kubevirt/testdata/affinity-no-values.yaml
index f304b2a6..7d01a42c 100644
--- a/pkg/cloudprovider/provider/kubevirt/testdata/affinity-no-values.yaml
+++ b/pkg/cloudprovider/provider/kubevirt/testdata/affinity-no-values.yaml
@@ -25,7 +25,7 @@ spec:
         source:
           http:
             url: http://x.y.z.t/ubuntu.img
-  running: true
+  runStrategy: Once
   template:
     metadata:
       labels:
diff --git a/pkg/cloudprovider/provider/kubevirt/testdata/affinity.yaml b/pkg/cloudprovider/provider/kubevirt/testdata/affinity.yaml
index 309e30d7..a28aded5 100644
--- a/pkg/cloudprovider/provider/kubevirt/testdata/affinity.yaml
+++ b/pkg/cloudprovider/provider/kubevirt/testdata/affinity.yaml
@@ -25,7 +25,7 @@ spec:
         source:
           http:
             url: http://x.y.z.t/ubuntu.img
-  running: true
+  runStrategy: Once
   template:
     metadata:
       labels:
diff --git a/pkg/cloudprovider/provider/kubevirt/testdata/custom-local-disk.yaml b/pkg/cloudprovider/provider/kubevirt/testdata/custom-local-disk.yaml
index e10d07c6..b77494b6 100644
--- a/pkg/cloudprovider/provider/kubevirt/testdata/custom-local-disk.yaml
+++ b/pkg/cloudprovider/provider/kubevirt/testdata/custom-local-disk.yaml
@@ -25,7 +25,7 @@ spec:
           pvc:
             namespace: ns
             name: dvname
-  running: true
+  runStrategy: Once
   template:
     metadata:
       creationTimestamp: null
diff --git a/pkg/cloudprovider/provider/kubevirt/testdata/http-image-source.yaml b/pkg/cloudprovider/provider/kubevirt/testdata/http-image-source.yaml
index 3bf7731e..caef1ed5 100644
--- a/pkg/cloudprovider/provider/kubevirt/testdata/http-image-source.yaml
+++ b/pkg/cloudprovider/provider/kubevirt/testdata/http-image-source.yaml
@@ -24,7 +24,7 @@ spec:
         source:
           http:
             url: http://x.y.z.t/ubuntu.img
-  running: true
+  runStrategy: Once
   template:
     metadata:
       creationTimestamp: null
diff --git a/pkg/cloudprovider/provider/kubevirt/testdata/instancetype-preference-custom.yaml b/pkg/cloudprovider/provider/kubevirt/testdata/instancetype-preference-custom.yaml
index 060171c4..ba699302 100644
--- a/pkg/cloudprovider/provider/kubevirt/testdata/instancetype-preference-custom.yaml
+++ b/pkg/cloudprovider/provider/kubevirt/testdata/instancetype-preference-custom.yaml
@@ -25,7 +25,7 @@ spec:
         source:
           http:
             url: http://x.y.z.t/ubuntu.img
-  running: true
+  runStrategy: Once
   instancetype:
     kind: VirtualMachineClusterInstancetype
     name: custom-it
diff --git a/pkg/cloudprovider/provider/kubevirt/testdata/instancetype-preference-standard.yaml b/pkg/cloudprovider/provider/kubevirt/testdata/instancetype-preference-standard.yaml
index b56229f6..1f54c87a 100644
--- a/pkg/cloudprovider/provider/kubevirt/testdata/instancetype-preference-standard.yaml
+++ b/pkg/cloudprovider/provider/kubevirt/testdata/instancetype-preference-standard.yaml
@@ -24,7 +24,7 @@ spec:
         source:
           http:
             url: http://x.y.z.t/ubuntu.img
-  running: true
+  runStrategy: Once
   instancetype:
     kind: VirtualMachineInstancetype
     name: standard-it
diff --git a/pkg/cloudprovider/provider/kubevirt/testdata/nominal-case.yaml b/pkg/cloudprovider/provider/kubevirt/testdata/nominal-case.yaml
index 5dddb4b5..24afa309 100644
--- a/pkg/cloudprovider/provider/kubevirt/testdata/nominal-case.yaml
+++ b/pkg/cloudprovider/provider/kubevirt/testdata/nominal-case.yaml
@@ -24,7 +24,7 @@ spec:
         source:
           http:
             url: http://x.y.z.t/ubuntu.img
-  running: true
+  runStrategy: Once
   template:
     metadata:
       creationTimestamp: null
diff --git a/pkg/cloudprovider/provider/kubevirt/testdata/pvc-image-source.yaml b/pkg/cloudprovider/provider/kubevirt/testdata/pvc-image-source.yaml
index 2cafabbd..978213ec 100644
--- a/pkg/cloudprovider/provider/kubevirt/testdata/pvc-image-source.yaml
+++ b/pkg/cloudprovider/provider/kubevirt/testdata/pvc-image-source.yaml
@@ -25,7 +25,7 @@ spec:
           pvc:
             namespace: ns
             name: dvname
-  running: true
+  runStrategy: Once
   template:
     metadata:
       creationTimestamp: null
diff --git a/pkg/cloudprovider/provider/kubevirt/testdata/secondary-disks.yaml b/pkg/cloudprovider/provider/kubevirt/testdata/secondary-disks.yaml
index 576172c9..0ce57a4b 100644
--- a/pkg/cloudprovider/provider/kubevirt/testdata/secondary-disks.yaml
+++ b/pkg/cloudprovider/provider/kubevirt/testdata/secondary-disks.yaml
@@ -50,7 +50,7 @@ spec:
         source:
           http:
             url: http://x.y.z.t/ubuntu.img
-  running: true
+  runStrategy: Once
   template:
     metadata:
       creationTimestamp: null
diff --git a/pkg/cloudprovider/provider/kubevirt/testdata/topologyspreadconstraints.yaml b/pkg/cloudprovider/provider/kubevirt/testdata/topologyspreadconstraints.yaml
index a36f6b84..36346072 100644
--- a/pkg/cloudprovider/provider/kubevirt/testdata/topologyspreadconstraints.yaml
+++ b/pkg/cloudprovider/provider/kubevirt/testdata/topologyspreadconstraints.yaml
@@ -24,7 +24,7 @@ spec:
         source:
           http:
             url: http://x.y.z.t/ubuntu.img
-  running: true
+  runStrategy: Once
   template:
     metadata:
       creationTimestamp: null
diff --git a/test/e2e/provisioning/testdata/machinedeployment-kubevirt.yaml b/test/e2e/provisioning/testdata/machinedeployment-kubevirt.yaml
index 90a46bfc..d97169ad 100644
--- a/test/e2e/provisioning/testdata/machinedeployment-kubevirt.yaml
+++ b/test/e2e/provisioning/testdata/machinedeployment-kubevirt.yaml
@@ -38,7 +38,7 @@ spec:
                 primaryDisk:
                   osImage: http://image-repo.kube-system.svc/images/<< KUBEVIRT_OS_IMAGE >>.img
                   size: "25Gi"
-                  storageClassName: px-csi-db
+                  storageClassName: rook-ceph-block
               dnsPolicy: "None"
               dnsConfig:
                 nameservers: