Repository: kubermatic/machine-controller
Pull request #1654 (2023-06-03 01:40:32)

[release/v1.54] Allow external CCMs to handle node objects before MC
**Manual backport of https://github.com/kubermatic/machine-controller/pull/1645**

**What this PR does / why we need it**:

Allow external CCMs to handle failing node objects before MC takes any action.
It prevents race condition between MC and CCM.

The PR introduces a `handleNodeFailuresWithExternalCCM` function that:

>handleNodeFailuresWithExternalCCM reacts to node status discovery of CCM's node lifecycle controller.
>If an instance at cloud provider is not found then it waits till CCM deletes node objects, that allows:
>   - create a new instance at cloud provider
>   - initialize a new node object - the object should not be reused between instance creation
>     for example, instance foo that got deleted and recreated should initialize a completely new node object
>     instead of reusing the old one as it can cause problems to update node's metadata, like IP address.
>
> If node is shut-down it allows MC to react accordingly to specific cloud provider requirements, those are:
>   - wait for node to become online again or
>   - delete a machine which cannot be recovered

**Which issue(s) this PR fixes**:
<!--optional, in `fixes #<issue number>` format, will close the issue(s) when PR gets merged-->
Fixes https://github.com/kubermatic/kubermatic/issues/12218

**What type of PR is this?**
/kind bug
<!--
Add one of the following kinds:
/kind bug
/kind cleanup
/kind documentation
/kind feature
/kind design

Optionally add one or more of the following kinds if applicable:
/kind api-change
/kind deprecation
/kind failing-test
/kind flake
/kind regression
/kind chore
-->

**Special notes for your reviewer**:

**Does this PR introduce a user-facing change? Then add your Release Note here**:
<!--
Write your release note:
1. Enter your extended release note in the below block. If the PR requires additional action from users switching to the new release, include the string "action required".
2. If no release note is required, just write "NONE".
-->
```release-note
Allow external CCMs to handle failing node objects before MC.
```

**Documentation**:
<!--
Please do one of the following options:
- Add a link to the existing documentation
- Add a link to the kubermatic/docs pull request
- If no documentation change is applicable then add:
  - TBD (documentation will be added later)
  - NONE (no documentation needed for this PR)
-->
```documentation
NONE
```


kubermatic-bot: [APPROVALNOTIFIER] This PR is **NOT APPROVED**

This pull-request has been approved by: *<a href="https://github.com/kubermatic/machine-controller/pull/1654#" title="Author self-approved">mfranczy</a>*
**Once this PR has been reviewed and has the lgtm label**, please assign ahmedwaleedmalik for approval. For more information see [the Kubernetes Code Review Process](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process).

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=kubermatic%2Fmachine-controller).

<details open>
Needs approval from an approver in each of these files:

- ~~[OWNERS](https://github.com/kubermatic/machine-controller/blob/release/v1.54/OWNERS)~~ [mfranczy]
- **[pkg/cloudprovider/provider/OWNERS](https://github.com/kubermatic/machine-controller/blob/release/v1.54/pkg/cloudprovider/provider/OWNERS)**

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":["ahmedwaleedmalik"]} -->
kubermatic-bot: @mfranczy: The following test **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-machine-controller-test | a15c246016ee91c8feaec0c35e01d86bdaa2b890 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1654/pull-machine-controller-test/1664809281582534656) | true | `/test pull-machine-controller-test`

[Full PR test history](https://public-prow.loodse.com/pr-history?org=kubermatic&repo=machine-controller&pr=1654)

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
kubermatic-bot: @mfranczy: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-machine-controller-test | a15c246016ee91c8feaec0c35e01d86bdaa2b890 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1654/pull-machine-controller-test/1664809281582534656) | true | `/test pull-machine-controller-test`
pull-machine-controller-build | a15c246016ee91c8feaec0c35e01d86bdaa2b890 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1654/pull-machine-controller-build/1664809277958656000) | true | `/test pull-machine-controller-build`

[Full PR test history](https://public-prow.loodse.com/pr-history?org=kubermatic&repo=machine-controller&pr=1654)

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
kubermatic-bot: @mfranczy: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-machine-controller-test | a15c246016ee91c8feaec0c35e01d86bdaa2b890 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1654/pull-machine-controller-test/1664809281582534656) | true | `/test pull-machine-controller-test`
pull-machine-controller-build | a15c246016ee91c8feaec0c35e01d86bdaa2b890 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1654/pull-machine-controller-build/1664809277958656000) | true | `/test pull-machine-controller-build`
pull-machine-controller-license-validation | a15c246016ee91c8feaec0c35e01d86bdaa2b890 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1654/pull-machine-controller-license-validation/1664809280743673856) | true | `/test pull-machine-controller-license-validation`

[Full PR test history](https://public-prow.loodse.com/pr-history?org=kubermatic&repo=machine-controller&pr=1654)

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
mfranczy: /retest
kubermatic-bot: @mfranczy: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-machine-controller-test | a15c246016ee91c8feaec0c35e01d86bdaa2b890 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1654/pull-machine-controller-test/1664809281582534656) | true | `/test pull-machine-controller-test`
pull-machine-controller-build | a15c246016ee91c8feaec0c35e01d86bdaa2b890 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1654/pull-machine-controller-build/1664809277958656000) | true | `/test pull-machine-controller-build`
pull-machine-controller-license-validation | a15c246016ee91c8feaec0c35e01d86bdaa2b890 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1654/pull-machine-controller-license-validation/1664809280743673856) | true | `/test pull-machine-controller-license-validation`
pull-machine-controller-e2e-aws-spot-instance | a15c246016ee91c8feaec0c35e01d86bdaa2b890 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1654/pull-machine-controller-e2e-aws-spot-instance/1664809277790883840) | true | `/test pull-machine-controller-e2e-aws-spot-instance`

[Full PR test history](https://public-prow.loodse.com/pr-history?org=kubermatic&repo=machine-controller&pr=1654)

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
kubermatic-bot: @mfranczy: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-machine-controller-test | a15c246016ee91c8feaec0c35e01d86bdaa2b890 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1654/pull-machine-controller-test/1664809281582534656) | true | `/test pull-machine-controller-test`
pull-machine-controller-build | a15c246016ee91c8feaec0c35e01d86bdaa2b890 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1654/pull-machine-controller-build/1664809277958656000) | true | `/test pull-machine-controller-build`
pull-machine-controller-license-validation | a15c246016ee91c8feaec0c35e01d86bdaa2b890 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1654/pull-machine-controller-license-validation/1664809280743673856) | true | `/test pull-machine-controller-license-validation`
pull-machine-controller-e2e-aws-spot-instance | a15c246016ee91c8feaec0c35e01d86bdaa2b890 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1654/pull-machine-controller-e2e-aws-spot-instance/1664809277790883840) | true | `/test pull-machine-controller-e2e-aws-spot-instance`
pull-machine-controller-lint | a15c246016ee91c8feaec0c35e01d86bdaa2b890 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1654/pull-machine-controller-lint/1664809278218702848) | true | `/test pull-machine-controller-lint`

[Full PR test history](https://public-prow.loodse.com/pr-history?org=kubermatic&repo=machine-controller&pr=1654)

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
kubermatic-bot: @mfranczy: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-machine-controller-test | a15c246016ee91c8feaec0c35e01d86bdaa2b890 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1654/pull-machine-controller-test/1664809281582534656) | true | `/test pull-machine-controller-test`
pull-machine-controller-build | a15c246016ee91c8feaec0c35e01d86bdaa2b890 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1654/pull-machine-controller-build/1664809277958656000) | true | `/test pull-machine-controller-build`
pull-machine-controller-e2e-aws-spot-instance | a15c246016ee91c8feaec0c35e01d86bdaa2b890 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1654/pull-machine-controller-e2e-aws-spot-instance/1664809277790883840) | true | `/test pull-machine-controller-e2e-aws-spot-instance`
pull-machine-controller-lint | a15c246016ee91c8feaec0c35e01d86bdaa2b890 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1654/pull-machine-controller-lint/1664809278218702848) | true | `/test pull-machine-controller-lint`
pull-machine-controller-license-validation | a15c246016ee91c8feaec0c35e01d86bdaa2b890 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1654/pull-machine-controller-license-validation/1664810182753914880) | true | `/test pull-machine-controller-license-validation`

[Full PR test history](https://public-prow.loodse.com/pr-history?org=kubermatic&repo=machine-controller&pr=1654)

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
kubermatic-bot: @mfranczy: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-machine-controller-test | a15c246016ee91c8feaec0c35e01d86bdaa2b890 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1654/pull-machine-controller-test/1664809281582534656) | true | `/test pull-machine-controller-test`
pull-machine-controller-build | a15c246016ee91c8feaec0c35e01d86bdaa2b890 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1654/pull-machine-controller-build/1664809277958656000) | true | `/test pull-machine-controller-build`
pull-machine-controller-e2e-aws-spot-instance | a15c246016ee91c8feaec0c35e01d86bdaa2b890 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1654/pull-machine-controller-e2e-aws-spot-instance/1664809277790883840) | true | `/test pull-machine-controller-e2e-aws-spot-instance`
pull-machine-controller-lint | a15c246016ee91c8feaec0c35e01d86bdaa2b890 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1654/pull-machine-controller-lint/1664809278218702848) | true | `/test pull-machine-controller-lint`
pull-machine-controller-license-validation | a15c246016ee91c8feaec0c35e01d86bdaa2b890 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1654/pull-machine-controller-license-validation/1664810182753914880) | true | `/test pull-machine-controller-license-validation`
pull-machine-controller-e2e-aws-legacy-userdata | a15c246016ee91c8feaec0c35e01d86bdaa2b890 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1654/pull-machine-controller-e2e-aws-legacy-userdata/1664809277723774976) | true | `/test pull-machine-controller-e2e-aws-legacy-userdata`

[Full PR test history](https://public-prow.loodse.com/pr-history?org=kubermatic&repo=machine-controller&pr=1654)

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
kubermatic-bot: @mfranczy: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-machine-controller-test | a15c246016ee91c8feaec0c35e01d86bdaa2b890 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1654/pull-machine-controller-test/1664809281582534656) | true | `/test pull-machine-controller-test`
pull-machine-controller-build | a15c246016ee91c8feaec0c35e01d86bdaa2b890 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1654/pull-machine-controller-build/1664809277958656000) | true | `/test pull-machine-controller-build`
pull-machine-controller-e2e-aws-spot-instance | a15c246016ee91c8feaec0c35e01d86bdaa2b890 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1654/pull-machine-controller-e2e-aws-spot-instance/1664809277790883840) | true | `/test pull-machine-controller-e2e-aws-spot-instance`
pull-machine-controller-lint | a15c246016ee91c8feaec0c35e01d86bdaa2b890 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1654/pull-machine-controller-lint/1664809278218702848) | true | `/test pull-machine-controller-lint`
pull-machine-controller-license-validation | a15c246016ee91c8feaec0c35e01d86bdaa2b890 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1654/pull-machine-controller-license-validation/1664810182753914880) | true | `/test pull-machine-controller-license-validation`
pull-machine-controller-e2e-aws-legacy-userdata | a15c246016ee91c8feaec0c35e01d86bdaa2b890 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1654/pull-machine-controller-e2e-aws-legacy-userdata/1664809277723774976) | true | `/test pull-machine-controller-e2e-aws-legacy-userdata`
pull-machine-controller-e2e-custom-ca | a15c246016ee91c8feaec0c35e01d86bdaa2b890 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1654/pull-machine-controller-e2e-custom-ca/1664809277589557248) | true | `/test pull-machine-controller-e2e-custom-ca`

[Full PR test history](https://public-prow.loodse.com/pr-history?org=kubermatic&repo=machine-controller&pr=1654)

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
mfranczy: /retest
kubermatic-bot: @mfranczy: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-machine-controller-test | a15c246016ee91c8feaec0c35e01d86bdaa2b890 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1654/pull-machine-controller-test/1664809281582534656) | true | `/test pull-machine-controller-test`
pull-machine-controller-build | a15c246016ee91c8feaec0c35e01d86bdaa2b890 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1654/pull-machine-controller-build/1664809277958656000) | true | `/test pull-machine-controller-build`
pull-machine-controller-e2e-aws-spot-instance | a15c246016ee91c8feaec0c35e01d86bdaa2b890 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1654/pull-machine-controller-e2e-aws-spot-instance/1664809277790883840) | true | `/test pull-machine-controller-e2e-aws-spot-instance`
pull-machine-controller-lint | a15c246016ee91c8feaec0c35e01d86bdaa2b890 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1654/pull-machine-controller-lint/1664809278218702848) | true | `/test pull-machine-controller-lint`
pull-machine-controller-license-validation | a15c246016ee91c8feaec0c35e01d86bdaa2b890 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1654/pull-machine-controller-license-validation/1664810182753914880) | true | `/test pull-machine-controller-license-validation`
pull-machine-controller-e2e-aws-legacy-userdata | a15c246016ee91c8feaec0c35e01d86bdaa2b890 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1654/pull-machine-controller-e2e-aws-legacy-userdata/1664809277723774976) | true | `/test pull-machine-controller-e2e-aws-legacy-userdata`
pull-machine-controller-e2e-custom-ca | a15c246016ee91c8feaec0c35e01d86bdaa2b890 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1654/pull-machine-controller-e2e-custom-ca/1664809277589557248) | true | `/test pull-machine-controller-e2e-custom-ca`
pull-machine-controller-e2e-invalid-objects-get-rejected | a15c246016ee91c8feaec0c35e01d86bdaa2b890 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1654/pull-machine-controller-e2e-invalid-objects-get-rejected/1664809277526642688) | true | `/test pull-machine-controller-e2e-invalid-objects-get-rejected`

[Full PR test history](https://public-prow.loodse.com/pr-history?org=kubermatic&repo=machine-controller&pr=1654)

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
kubermatic-bot: @mfranczy: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-machine-controller-build | a15c246016ee91c8feaec0c35e01d86bdaa2b890 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1654/pull-machine-controller-build/1664809277958656000) | true | `/test pull-machine-controller-build`
pull-machine-controller-e2e-aws-spot-instance | a15c246016ee91c8feaec0c35e01d86bdaa2b890 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1654/pull-machine-controller-e2e-aws-spot-instance/1664809277790883840) | true | `/test pull-machine-controller-e2e-aws-spot-instance`
pull-machine-controller-lint | a15c246016ee91c8feaec0c35e01d86bdaa2b890 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1654/pull-machine-controller-lint/1664809278218702848) | true | `/test pull-machine-controller-lint`
pull-machine-controller-license-validation | a15c246016ee91c8feaec0c35e01d86bdaa2b890 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1654/pull-machine-controller-license-validation/1664810182753914880) | true | `/test pull-machine-controller-license-validation`
pull-machine-controller-e2e-aws-legacy-userdata | a15c246016ee91c8feaec0c35e01d86bdaa2b890 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1654/pull-machine-controller-e2e-aws-legacy-userdata/1664809277723774976) | true | `/test pull-machine-controller-e2e-aws-legacy-userdata`
pull-machine-controller-e2e-custom-ca | a15c246016ee91c8feaec0c35e01d86bdaa2b890 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1654/pull-machine-controller-e2e-custom-ca/1664809277589557248) | true | `/test pull-machine-controller-e2e-custom-ca`
pull-machine-controller-e2e-invalid-objects-get-rejected | a15c246016ee91c8feaec0c35e01d86bdaa2b890 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1654/pull-machine-controller-e2e-invalid-objects-get-rejected/1664809277526642688) | true | `/test pull-machine-controller-e2e-invalid-objects-get-rejected`
pull-machine-controller-e2e-ubuntu-upgrade | a15c246016ee91c8feaec0c35e01d86bdaa2b890 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1654/pull-machine-controller-e2e-ubuntu-upgrade/1664809277660860416) | true | `/test pull-machine-controller-e2e-ubuntu-upgrade`

[Full PR test history](https://public-prow.loodse.com/pr-history?org=kubermatic&repo=machine-controller&pr=1654)

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
kubermatic-bot: @mfranczy: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-machine-controller-build | a15c246016ee91c8feaec0c35e01d86bdaa2b890 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1654/pull-machine-controller-build/1664809277958656000) | true | `/test pull-machine-controller-build`
pull-machine-controller-e2e-aws-spot-instance | a15c246016ee91c8feaec0c35e01d86bdaa2b890 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1654/pull-machine-controller-e2e-aws-spot-instance/1664809277790883840) | true | `/test pull-machine-controller-e2e-aws-spot-instance`
pull-machine-controller-lint | a15c246016ee91c8feaec0c35e01d86bdaa2b890 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1654/pull-machine-controller-lint/1664809278218702848) | true | `/test pull-machine-controller-lint`
pull-machine-controller-license-validation | a15c246016ee91c8feaec0c35e01d86bdaa2b890 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1654/pull-machine-controller-license-validation/1664810182753914880) | true | `/test pull-machine-controller-license-validation`
pull-machine-controller-e2e-aws-legacy-userdata | a15c246016ee91c8feaec0c35e01d86bdaa2b890 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1654/pull-machine-controller-e2e-aws-legacy-userdata/1664809277723774976) | true | `/test pull-machine-controller-e2e-aws-legacy-userdata`
pull-machine-controller-e2e-custom-ca | a15c246016ee91c8feaec0c35e01d86bdaa2b890 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1654/pull-machine-controller-e2e-custom-ca/1664809277589557248) | true | `/test pull-machine-controller-e2e-custom-ca`
pull-machine-controller-e2e-invalid-objects-get-rejected | a15c246016ee91c8feaec0c35e01d86bdaa2b890 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1654/pull-machine-controller-e2e-invalid-objects-get-rejected/1664809277526642688) | true | `/test pull-machine-controller-e2e-invalid-objects-get-rejected`
pull-machine-controller-e2e-ubuntu-upgrade | a15c246016ee91c8feaec0c35e01d86bdaa2b890 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1654/pull-machine-controller-e2e-ubuntu-upgrade/1664809277660860416) | true | `/test pull-machine-controller-e2e-ubuntu-upgrade`
pull-machine-controller-e2e-openstack-project-auth | a15c246016ee91c8feaec0c35e01d86bdaa2b890 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1654/pull-machine-controller-e2e-openstack-project-auth/1664809277908324352) | true | `/test pull-machine-controller-e2e-openstack-project-auth`

[Full PR test history](https://public-prow.loodse.com/pr-history?org=kubermatic&repo=machine-controller&pr=1654)

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
kubermatic-bot: @mfranczy: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-machine-controller-e2e-aws-spot-instance | a15c246016ee91c8feaec0c35e01d86bdaa2b890 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1654/pull-machine-controller-e2e-aws-spot-instance/1664809277790883840) | true | `/test pull-machine-controller-e2e-aws-spot-instance`
pull-machine-controller-lint | a15c246016ee91c8feaec0c35e01d86bdaa2b890 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1654/pull-machine-controller-lint/1664809278218702848) | true | `/test pull-machine-controller-lint`
pull-machine-controller-license-validation | a15c246016ee91c8feaec0c35e01d86bdaa2b890 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1654/pull-machine-controller-license-validation/1664810182753914880) | true | `/test pull-machine-controller-license-validation`
pull-machine-controller-e2e-aws-legacy-userdata | a15c246016ee91c8feaec0c35e01d86bdaa2b890 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1654/pull-machine-controller-e2e-aws-legacy-userdata/1664809277723774976) | true | `/test pull-machine-controller-e2e-aws-legacy-userdata`
pull-machine-controller-e2e-custom-ca | a15c246016ee91c8feaec0c35e01d86bdaa2b890 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1654/pull-machine-controller-e2e-custom-ca/1664809277589557248) | true | `/test pull-machine-controller-e2e-custom-ca`
pull-machine-controller-e2e-ubuntu-upgrade | a15c246016ee91c8feaec0c35e01d86bdaa2b890 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1654/pull-machine-controller-e2e-ubuntu-upgrade/1664809277660860416) | true | `/test pull-machine-controller-e2e-ubuntu-upgrade`
pull-machine-controller-e2e-openstack-project-auth | a15c246016ee91c8feaec0c35e01d86bdaa2b890 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1654/pull-machine-controller-e2e-openstack-project-auth/1664809277908324352) | true | `/test pull-machine-controller-e2e-openstack-project-auth`
pull-machine-controller-e2e-invalid-objects-get-rejected | a15c246016ee91c8feaec0c35e01d86bdaa2b890 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1654/pull-machine-controller-e2e-invalid-objects-get-rejected/1664811614185984000) | true | `/test pull-machine-controller-e2e-invalid-objects-get-rejected`

[Full PR test history](https://public-prow.loodse.com/pr-history?org=kubermatic&repo=machine-controller&pr=1654)

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
kubermatic-bot: @mfranczy: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-machine-controller-e2e-aws-spot-instance | a15c246016ee91c8feaec0c35e01d86bdaa2b890 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1654/pull-machine-controller-e2e-aws-spot-instance/1664809277790883840) | true | `/test pull-machine-controller-e2e-aws-spot-instance`
pull-machine-controller-lint | a15c246016ee91c8feaec0c35e01d86bdaa2b890 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1654/pull-machine-controller-lint/1664809278218702848) | true | `/test pull-machine-controller-lint`
pull-machine-controller-e2e-custom-ca | a15c246016ee91c8feaec0c35e01d86bdaa2b890 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1654/pull-machine-controller-e2e-custom-ca/1664809277589557248) | true | `/test pull-machine-controller-e2e-custom-ca`
pull-machine-controller-e2e-ubuntu-upgrade | a15c246016ee91c8feaec0c35e01d86bdaa2b890 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1654/pull-machine-controller-e2e-ubuntu-upgrade/1664809277660860416) | true | `/test pull-machine-controller-e2e-ubuntu-upgrade`
pull-machine-controller-e2e-openstack-project-auth | a15c246016ee91c8feaec0c35e01d86bdaa2b890 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1654/pull-machine-controller-e2e-openstack-project-auth/1664809277908324352) | true | `/test pull-machine-controller-e2e-openstack-project-auth`
pull-machine-controller-e2e-invalid-objects-get-rejected | a15c246016ee91c8feaec0c35e01d86bdaa2b890 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1654/pull-machine-controller-e2e-invalid-objects-get-rejected/1664811614185984000) | true | `/test pull-machine-controller-e2e-invalid-objects-get-rejected`
pull-machine-controller-e2e-aws-legacy-userdata | a15c246016ee91c8feaec0c35e01d86bdaa2b890 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1654/pull-machine-controller-e2e-aws-legacy-userdata/1664811614303424512) | true | `/test pull-machine-controller-e2e-aws-legacy-userdata`

[Full PR test history](https://public-prow.loodse.com/pr-history?org=kubermatic&repo=machine-controller&pr=1654)

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
kubermatic-bot: @mfranczy: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-machine-controller-e2e-aws-spot-instance | a15c246016ee91c8feaec0c35e01d86bdaa2b890 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1654/pull-machine-controller-e2e-aws-spot-instance/1664809277790883840) | true | `/test pull-machine-controller-e2e-aws-spot-instance`
pull-machine-controller-lint | a15c246016ee91c8feaec0c35e01d86bdaa2b890 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1654/pull-machine-controller-lint/1664809278218702848) | true | `/test pull-machine-controller-lint`
pull-machine-controller-e2e-ubuntu-upgrade | a15c246016ee91c8feaec0c35e01d86bdaa2b890 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1654/pull-machine-controller-e2e-ubuntu-upgrade/1664809277660860416) | true | `/test pull-machine-controller-e2e-ubuntu-upgrade`
pull-machine-controller-e2e-openstack-project-auth | a15c246016ee91c8feaec0c35e01d86bdaa2b890 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1654/pull-machine-controller-e2e-openstack-project-auth/1664809277908324352) | true | `/test pull-machine-controller-e2e-openstack-project-auth`
pull-machine-controller-e2e-invalid-objects-get-rejected | a15c246016ee91c8feaec0c35e01d86bdaa2b890 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1654/pull-machine-controller-e2e-invalid-objects-get-rejected/1664811614185984000) | true | `/test pull-machine-controller-e2e-invalid-objects-get-rejected`
pull-machine-controller-e2e-aws-legacy-userdata | a15c246016ee91c8feaec0c35e01d86bdaa2b890 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1654/pull-machine-controller-e2e-aws-legacy-userdata/1664811614303424512) | true | `/test pull-machine-controller-e2e-aws-legacy-userdata`
pull-machine-controller-e2e-custom-ca | a15c246016ee91c8feaec0c35e01d86bdaa2b890 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1654/pull-machine-controller-e2e-custom-ca/1664811614244704256) | true | `/test pull-machine-controller-e2e-custom-ca`

[Full PR test history](https://public-prow.loodse.com/pr-history?org=kubermatic&repo=machine-controller&pr=1654)

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
kubermatic-bot: @mfranczy: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-machine-controller-e2e-ubuntu-upgrade | a15c246016ee91c8feaec0c35e01d86bdaa2b890 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1654/pull-machine-controller-e2e-ubuntu-upgrade/1664809277660860416) | true | `/test pull-machine-controller-e2e-ubuntu-upgrade`
pull-machine-controller-e2e-openstack-project-auth | a15c246016ee91c8feaec0c35e01d86bdaa2b890 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1654/pull-machine-controller-e2e-openstack-project-auth/1664809277908324352) | true | `/test pull-machine-controller-e2e-openstack-project-auth`
pull-machine-controller-e2e-invalid-objects-get-rejected | a15c246016ee91c8feaec0c35e01d86bdaa2b890 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1654/pull-machine-controller-e2e-invalid-objects-get-rejected/1664811614185984000) | true | `/test pull-machine-controller-e2e-invalid-objects-get-rejected`
pull-machine-controller-e2e-aws-legacy-userdata | a15c246016ee91c8feaec0c35e01d86bdaa2b890 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1654/pull-machine-controller-e2e-aws-legacy-userdata/1664811614303424512) | true | `/test pull-machine-controller-e2e-aws-legacy-userdata`
pull-machine-controller-e2e-custom-ca | a15c246016ee91c8feaec0c35e01d86bdaa2b890 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1654/pull-machine-controller-e2e-custom-ca/1664811614244704256) | true | `/test pull-machine-controller-e2e-custom-ca`
pull-machine-controller-e2e-aws-spot-instance | a15c246016ee91c8feaec0c35e01d86bdaa2b890 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1654/pull-machine-controller-e2e-aws-spot-instance/1664811614357950464) | true | `/test pull-machine-controller-e2e-aws-spot-instance`

[Full PR test history](https://public-prow.loodse.com/pr-history?org=kubermatic&repo=machine-controller&pr=1654)

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
kubermatic-bot: @mfranczy: The following tests **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
pull-machine-controller-e2e-ubuntu-upgrade | a15c246016ee91c8feaec0c35e01d86bdaa2b890 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1654/pull-machine-controller-e2e-ubuntu-upgrade/1664809277660860416) | true | `/test pull-machine-controller-e2e-ubuntu-upgrade`
pull-machine-controller-e2e-openstack-project-auth | a15c246016ee91c8feaec0c35e01d86bdaa2b890 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1654/pull-machine-controller-e2e-openstack-project-auth/1664809277908324352) | true | `/test pull-machine-controller-e2e-openstack-project-auth`
pull-machine-controller-e2e-invalid-objects-get-rejected | a15c246016ee91c8feaec0c35e01d86bdaa2b890 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1654/pull-machine-controller-e2e-invalid-objects-get-rejected/1664811614185984000) | true | `/test pull-machine-controller-e2e-invalid-objects-get-rejected`
pull-machine-controller-e2e-aws-legacy-userdata | a15c246016ee91c8feaec0c35e01d86bdaa2b890 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1654/pull-machine-controller-e2e-aws-legacy-userdata/1664811614303424512) | true | `/test pull-machine-controller-e2e-aws-legacy-userdata`
pull-machine-controller-e2e-custom-ca | a15c246016ee91c8feaec0c35e01d86bdaa2b890 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1654/pull-machine-controller-e2e-custom-ca/1664811614244704256) | true | `/test pull-machine-controller-e2e-custom-ca`
pull-machine-controller-e2e-aws-spot-instance | a15c246016ee91c8feaec0c35e01d86bdaa2b890 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1654/pull-machine-controller-e2e-aws-spot-instance/1664811614357950464) | true | `/test pull-machine-controller-e2e-aws-spot-instance`
pull-machine-controller-e2e-kubevirt | a15c246016ee91c8feaec0c35e01d86bdaa2b890 | [link](https://public-prow.loodse.com/view/gs/prow-dev-public-data/pr-logs/pull/kubermatic_machine-controller/1654/pull-machine-controller-e2e-kubevirt/1664813514599960576) | true | `/test pull-machine-controller-e2e-kubevirt`

[Full PR test history](https://public-prow.loodse.com/pr-history?org=kubermatic&repo=machine-controller&pr=1654)

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->

Patch:
diff --git a/go.mod b/go.mod
index 890c1464..f3e6a6e9 100644
--- a/go.mod
+++ b/go.mod
@@ -40,7 +40,7 @@ require (
 	golang.org/x/oauth2 v0.0.0-20220608161450-d0670ef3b1eb
 	gomodules.xyz/jsonpatch/v2 v2.2.0
 	google.golang.org/api v0.74.0
-	google.golang.org/grpc v1.45.0
+	google.golang.org/grpc v1.47.0
 	gopkg.in/gcfg.v1 v1.2.3
 	gopkg.in/yaml.v3 v3.0.1
 	// Please ensure that you update the image tags in `examples/operating-system-manager.yaml` as well.
@@ -49,6 +49,7 @@ require (
 	k8s.io/apiextensions-apiserver v0.24.2
 	k8s.io/apimachinery v0.24.2
 	k8s.io/client-go v12.0.0+incompatible
+	k8s.io/cloud-provider v0.24.2
 	k8s.io/klog v1.0.0
 	k8s.io/kubelet v0.24.2
 	k8s.io/utils v0.0.0-20220210201930-3a6ce19ff2f9
@@ -143,7 +144,7 @@ require (
 	golang.org/x/text v0.4.0 // indirect
 	golang.org/x/time v0.0.0-20220609170525-579cf78fd858 // indirect
 	google.golang.org/appengine v1.6.7 // indirect
-	google.golang.org/genproto v0.0.0-20220413183235-5e96e2839df9 // indirect
+	google.golang.org/genproto v0.0.0-20220502173005-c8bf987b8c21 // indirect
 	google.golang.org/protobuf v1.28.0 // indirect
 	gopkg.in/DATA-DOG/go-sqlmock.v1 v1.3.0 // indirect
 	gopkg.in/inf.v0 v0.9.1 // indirect
diff --git a/go.sum b/go.sum
index 3dbf4460..708b9e02 100644
--- a/go.sum
+++ b/go.sum
@@ -175,6 +175,7 @@ github.com/cncf/udpa/go v0.0.0-20210930031921-04548b0d99d4/go.mod h1:6pvJx4me5XP
 github.com/cncf/xds/go v0.0.0-20210312221358-fbca930ec8ed/go.mod h1:eXthEFrGJvWHgFFCl3hGmgk+/aYT6PnTQLykKQRLhEs=
 github.com/cncf/xds/go v0.0.0-20210805033703-aa0b78936158/go.mod h1:eXthEFrGJvWHgFFCl3hGmgk+/aYT6PnTQLykKQRLhEs=
 github.com/cncf/xds/go v0.0.0-20210922020428-25de7278fc84/go.mod h1:eXthEFrGJvWHgFFCl3hGmgk+/aYT6PnTQLykKQRLhEs=
+github.com/cncf/xds/go v0.0.0-20211001041855-01bcc9b48dfe/go.mod h1:eXthEFrGJvWHgFFCl3hGmgk+/aYT6PnTQLykKQRLhEs=
 github.com/cncf/xds/go v0.0.0-20211011173535-cb28da3451f1/go.mod h1:eXthEFrGJvWHgFFCl3hGmgk+/aYT6PnTQLykKQRLhEs=
 github.com/cockroachdb/datadriven v0.0.0-20190809214429-80d97fb3cbaa/go.mod h1:zn76sxSg3SzpJ0PPJaLDCu+Bu0Lg3sKTORVIj19EIF8=
 github.com/cockroachdb/datadriven v0.0.0-20200714090401-bf6692d28da5/go.mod h1:h6jFvWxBdQXxjopDMZyH2UVceIRfR84bdzbkoKrsWNo=
@@ -255,6 +256,7 @@ github.com/envoyproxy/go-control-plane v0.9.9-0.20201210154907-fd9021fe5dad/go.m
 github.com/envoyproxy/go-control-plane v0.9.9-0.20210217033140-668b12f5399d/go.mod h1:cXg6YxExXjJnVBQHBLXeUAgxn2UodCpnH306RInaBQk=
 github.com/envoyproxy/go-control-plane v0.9.9-0.20210512163311-63b5d3c536b0/go.mod h1:hliV/p42l8fGbc6Y9bQ70uLwIvmJyVE5k4iMKlh8wCQ=
 github.com/envoyproxy/go-control-plane v0.9.10-0.20210907150352-cf90f659a021/go.mod h1:AFq3mo9L8Lqqiid3OhADV3RfLJnjiw63cSpi+fDTRC0=
+github.com/envoyproxy/go-control-plane v0.10.2-0.20220325020618-49ff273808a1/go.mod h1:KJwIaB5Mv44NWtYuAOFCVOjcI94vtpEz2JU/D2v6IjE=
 github.com/envoyproxy/protoc-gen-validate v0.1.0/go.mod h1:iSmxcyjqTsJpI2R4NaDN7+kN2VEUnK/pcBlmesArF7c=
 github.com/evanphx/json-patch v0.5.2/go.mod h1:ZWS5hhDbVDyob71nXKNL0+PWn6ToqBHMikGIFbs31qQ=
 github.com/evanphx/json-patch v4.12.0+incompatible/go.mod h1:50XU6AFN0ol/bzJsmQLiYLvXMP4fmwYFNcr97nuDLSk=
@@ -1407,8 +1409,9 @@ google.golang.org/genproto v0.0.0-20220304144024-325a89244dc8/go.mod h1:kGP+zUP2
 google.golang.org/genproto v0.0.0-20220310185008-1973136f34c6/go.mod h1:kGP+zUP2Ddo0ayMi4YuN7C3WZyJvGLZRh8Z5wnAqvEI=
 google.golang.org/genproto v0.0.0-20220324131243-acbaeb5b85eb/go.mod h1:hAL49I2IFola2sVEjAn7MEwsja0xp51I0tlGAf9hz4E=
 google.golang.org/genproto v0.0.0-20220407144326-9054f6ed7bac/go.mod h1:8w6bsBMX6yCPbAVTeqQHvzxW0EIFigd5lZyahWgyfDo=
-google.golang.org/genproto v0.0.0-20220413183235-5e96e2839df9 h1:XGQ6tc+EnM35IAazg4y6AHmUg4oK8NXsXaILte1vRlk=
 google.golang.org/genproto v0.0.0-20220413183235-5e96e2839df9/go.mod h1:8w6bsBMX6yCPbAVTeqQHvzxW0EIFigd5lZyahWgyfDo=
+google.golang.org/genproto v0.0.0-20220502173005-c8bf987b8c21 h1:hrbNEivu7Zn1pxvHk6MBrq9iE22woVILTHqexqBxe6I=
+google.golang.org/genproto v0.0.0-20220502173005-c8bf987b8c21/go.mod h1:RAyBrSAP7Fh3Nc84ghnVLDPuV51xc9agzmm4Ph6i0Q4=
 google.golang.org/grpc v1.17.0/go.mod h1:6QZJwpn2B+Zp71q/5VxRsJ6NXXVCE5NRUHRo+f3cWCs=
 google.golang.org/grpc v1.19.0/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c=
 google.golang.org/grpc v1.20.0/go.mod h1:chYK+tFQF0nDUGJgXMSgLCQk3phJEuONr2DCgLDdAQM=
@@ -1443,8 +1446,10 @@ google.golang.org/grpc v1.39.1/go.mod h1:PImNr+rS9TWYb2O4/emRugxiyHZ5JyHW5F+RPnD
 google.golang.org/grpc v1.40.0/go.mod h1:ogyxbiOoUXAkP+4+xa6PZSE9DZgIHtSpzjDTB9KAK34=
 google.golang.org/grpc v1.40.1/go.mod h1:ogyxbiOoUXAkP+4+xa6PZSE9DZgIHtSpzjDTB9KAK34=
 google.golang.org/grpc v1.44.0/go.mod h1:k+4IHHFw41K8+bbowsex27ge2rCb65oeWqe4jJ590SU=
-google.golang.org/grpc v1.45.0 h1:NEpgUqV3Z+ZjkqMsxMg11IaDrXY4RY6CQukSGK0uI1M=
 google.golang.org/grpc v1.45.0/go.mod h1:lN7owxKUQEqMfSyQikvvk5tf/6zMPsrK+ONuO11+0rQ=
+google.golang.org/grpc v1.46.0/go.mod h1:vN9eftEi1UMyUsIF80+uQXhHjbXYbm0uXoFCACuMGWk=
+google.golang.org/grpc v1.47.0 h1:9n77onPX5F3qfFCqjy9dhn8PbNQsIKeVU04J9G7umt8=
+google.golang.org/grpc v1.47.0/go.mod h1:vN9eftEi1UMyUsIF80+uQXhHjbXYbm0uXoFCACuMGWk=
 google.golang.org/grpc/cmd/protoc-gen-go-grpc v1.1.0/go.mod h1:6Kw0yEErY5E/yWrBtf03jp27GLLJujG4z/JK95pnjjw=
 google.golang.org/protobuf v0.0.0-20200109180630-ec00e32a8dfd/go.mod h1:DFci5gLYBciE7Vtevhsrf46CRTquxDuWsQurQQe4oz8=
 google.golang.org/protobuf v0.0.0-20200221191635-4d8936d0db64/go.mod h1:kwYJMbMJ01Woi6D6+Kah6886xMZcty6N08ah7+eCXa0=
@@ -1532,10 +1537,14 @@ k8s.io/apimachinery v0.24.2/go.mod h1:82Bi4sCzVBdpYjyI4jY6aHX+YCUchUIrZrXKedjd2U
 k8s.io/apiserver v0.24.2/go.mod h1:pSuKzr3zV+L+MWqsEo0kHHYwCo77AT5qXbFXP2jbvFI=
 k8s.io/client-go v0.24.2 h1:CoXFSf8if+bLEbinDqN9ePIDGzcLtqhfd6jpfnwGOFA=
 k8s.io/client-go v0.24.2/go.mod h1:zg4Xaoo+umDsfCWr4fCnmLEtQXyCNXCvJuSsglNcV30=
+k8s.io/cloud-provider v0.24.2 h1:DYNf90zS/GAQbEHsTfJsH4Oas7vim4U+WU9GftMQlfs=
+k8s.io/cloud-provider v0.24.2/go.mod h1:a7jyWjizk+IKbcIf8+mX2cj3NvpRv9ZyGdXDyb8UEkI=
 k8s.io/code-generator v0.23.3/go.mod h1:S0Q1JVA+kSzTI1oUvbKAxZY/DYbA/ZUb4Uknog12ETk=
 k8s.io/code-generator v0.24.2/go.mod h1:dpVhs00hTuTdTY6jvVxvTFCk6gSMrtfRydbhZwHI15w=
 k8s.io/component-base v0.24.2 h1:kwpQdoSfbcH+8MPN4tALtajLDfSfYxBDYlXobNWI6OU=
 k8s.io/component-base v0.24.2/go.mod h1:ucHwW76dajvQ9B7+zecZAP3BVqvrHoOxm8olHEg0nmM=
+k8s.io/component-helpers v0.24.2/go.mod h1:TRQPBQKfmqkmV6c0HAmUs8cXVNYYYLsXy4zu8eODi9g=
+k8s.io/controller-manager v0.24.2/go.mod h1:hpwCof4KxP4vrw/M5QiVxU6Zmmggmr1keGXtjGHF+vc=
 k8s.io/gengo v0.0.0-20210813121822-485abfe95c7c/go.mod h1:FiNAH4ZV3gBg2Kwh89tzAEV2be7d5xI0vBa/VySYy3E=
 k8s.io/gengo v0.0.0-20211129171323-c02415ce4185/go.mod h1:FiNAH4ZV3gBg2Kwh89tzAEV2be7d5xI0vBa/VySYy3E=
 k8s.io/klog v1.0.0 h1:Pt+yjF5aB1xDSVbau4VsWe+dQNzA0qv1LlXdC2dF6Q8=
diff --git a/hack/ci/setup-kind-cluster.sh b/hack/ci/setup-kind-cluster.sh
index b30c7882..7cc2c9a3 100755
--- a/hack/ci/setup-kind-cluster.sh
+++ b/hack/ci/setup-kind-cluster.sh
@@ -157,7 +157,7 @@ if [ -z "${DISABLE_CLUSTER_EXPOSER:-}" ]; then
     cd /tmp/kubermatic
     echodate "Cloning cluster exposer"
     KKP_REPO_URL="${KKP_REPO_URL:-https://github.com/kubermatic/kubermatic.git}"
-    KKP_REPO_TAG="${KKP_REPO_BRANCH:-main}"
+    KKP_REPO_TAG="${KKP_REPO_BRANCH:-release/v2.21}"
     git clone --depth 1 --branch "${KKP_REPO_TAG}" "${KKP_REPO_URL}" .
 
     echodate "Building cluster exposer"
diff --git a/pkg/cloudprovider/provider/anexia/provider.go b/pkg/cloudprovider/provider/anexia/provider.go
index 0032c972..d57505ff 100644
--- a/pkg/cloudprovider/provider/anexia/provider.go
+++ b/pkg/cloudprovider/provider/anexia/provider.go
@@ -166,16 +166,19 @@ func provisionVM(ctx context.Context, client anxclient.Client) error {
 
 		vm.Script = base64.StdEncoding.EncodeToString([]byte(reconcileContext.UserData))
 
-		for index, dnsServer := range reconcileContext.ProviderConfig.Network.DNS.Servers {
-			switch index {
-			case 0:
-				vm.DNS1 = dnsServer
-			case 1:
-				vm.DNS2 = dnsServer
-			case 2:
-				vm.DNS3 = dnsServer
-			case 3:
-				vm.DNS4 = dnsServer
+		providerCfg := reconcileContext.ProviderConfig
+		if providerCfg.Network != nil {
+			for index, dnsServer := range providerCfg.Network.DNS.Servers {
+				switch index {
+				case 0:
+					vm.DNS1 = dnsServer
+				case 1:
+					vm.DNS2 = dnsServer
+				case 2:
+					vm.DNS3 = dnsServer
+				case 3:
+					vm.DNS4 = dnsServer
+				}
 			}
 		}
 
diff --git a/pkg/cloudprovider/provider/kubevirt/provider.go b/pkg/cloudprovider/provider/kubevirt/provider.go
index 2523c62f..5ad61c44 100644
--- a/pkg/cloudprovider/provider/kubevirt/provider.go
+++ b/pkg/cloudprovider/provider/kubevirt/provider.go
@@ -394,18 +394,6 @@ func (p *provider) Get(ctx context.Context, machine *clusterv1alpha1.Machine, _
 		return nil, cloudprovidererrors.ErrInstanceNotFound
 	}
 
-	if virtualMachineInstance.Status.Phase == kubevirtv1.Failed ||
-		// The VMI enters phase succeeded if someone issues a kubectl
-		// delete pod on the virt-launcher pod it runs in
-		virtualMachineInstance.Status.Phase == kubevirtv1.Succeeded {
-		// The pod got deleted, delete the VMI and return ErrNotFound so the VMI
-		// will get recreated
-		if err := sigClient.Delete(ctx, virtualMachineInstance); err != nil {
-			return nil, fmt.Errorf("failed to delete failed VMI %s: %w", machine.Name, err)
-		}
-		return nil, cloudprovidererrors.ErrInstanceNotFound
-	}
-
 	return &kubeVirtServer{vmi: *virtualMachineInstance}, nil
 }
 
@@ -543,6 +531,8 @@ func (p *provider) Create(ctx context.Context, machine *clusterv1alpha1.Machine,
 		return nil, fmt.Errorf("could not compute a random MAC address")
 	}
 
+	runStrategyOnce := kubevirtv1.RunStrategyOnce
+
 	virtualMachine := &kubevirtv1.VirtualMachine{
 		ObjectMeta: metav1.ObjectMeta{
 			Name:      machine.Name,
@@ -550,7 +540,7 @@ func (p *provider) Create(ctx context.Context, machine *clusterv1alpha1.Machine,
 			Labels:    labels,
 		},
 		Spec: kubevirtv1.VirtualMachineSpec{
-			Running: utilpointer.BoolPtr(true),
+			RunStrategy: &runStrategyOnce,
 			Template: &kubevirtv1.VirtualMachineInstanceTemplateSpec{
 				ObjectMeta: metav1.ObjectMeta{
 					Annotations: annotations,
@@ -614,7 +604,6 @@ func (p *provider) Cleanup(ctx context.Context, machine *clusterv1alpha1.Machine
 		if !kerrors.IsNotFound(err) {
 			return false, fmt.Errorf("failed to get VirtualMachineInstance %s: %w", machine.Name, err)
 		}
-		// VMI is gone
 		return true, nil
 	}
 
diff --git a/pkg/controller/machine/machine_controller.go b/pkg/controller/machine/machine_controller.go
index d65b8782..1a4d026d 100644
--- a/pkg/controller/machine/machine_controller.go
+++ b/pkg/controller/machine/machine_controller.go
@@ -64,6 +64,7 @@ import (
 	"k8s.io/client-go/tools/record"
 	"k8s.io/client-go/tools/reference"
 	"k8s.io/client-go/util/retry"
+	ccmapi "k8s.io/cloud-provider/api"
 	"k8s.io/klog"
 	"sigs.k8s.io/controller-runtime/pkg/client"
 	ctrlruntimeclient "sigs.k8s.io/controller-runtime/pkg/client"
@@ -419,7 +420,8 @@ func (r *Reconciler) reconcile(ctx context.Context, machine *clusterv1alpha1.Mac
 
 	// step 2: check if a user requested to delete the machine
 	if machine.DeletionTimestamp != nil {
-		return r.deleteMachine(ctx, prov, providerConfig.CloudProvider, machine)
+		skipEviction := false
+		return r.deleteMachine(ctx, prov, providerConfig.CloudProvider, machine, skipEviction)
 	}
 
 	// Step 3: Essentially creates an instance for the given machine.
@@ -453,6 +455,9 @@ func (r *Reconciler) reconcile(ctx context.Context, machine *clusterv1alpha1.Mac
 		}
 	} else {
 		// Node is not ready anymore? Maybe it got deleted
+		if r.nodeSettings.ExternalCloudProvider {
+			return r.handleNodeFailuresWithExternalCCM(ctx, prov, providerConfig, node, machine)
+		}
 		return r.ensureInstanceExistsForMachine(ctx, prov, machine, userdataPlugin, providerConfig)
 	}
 
@@ -504,7 +509,7 @@ func (r *Reconciler) shouldCleanupVolumes(ctx context.Context, machine *clusterv
 func (r *Reconciler) shouldEvict(ctx context.Context, machine *clusterv1alpha1.Machine) (bool, error) {
 	// If the deletion got triggered a few hours ago, skip eviction.
 	// We assume here that the eviction is blocked by misconfiguration or a misbehaving kubelet and/or controller-runtime
-	if time.Since(machine.DeletionTimestamp.Time) > r.skipEvictionAfter {
+	if machine.DeletionTimestamp != nil && time.Since(machine.DeletionTimestamp.Time) > r.skipEvictionAfter {
 		klog.V(0).Infof("Skipping eviction for machine %q since the deletion got triggered %.2f minutes ago", machine.Name, r.skipEvictionAfter.Minutes())
 		return false, nil
 	}
@@ -559,11 +564,25 @@ func (r *Reconciler) shouldEvict(ctx context.Context, machine *clusterv1alpha1.M
 }
 
 // deleteMachine makes sure that an instance has gone in a series of steps.
-func (r *Reconciler) deleteMachine(ctx context.Context, prov cloudprovidertypes.Provider, providerName providerconfigtypes.CloudProvider, machine *clusterv1alpha1.Machine) (*reconcile.Result, error) {
-	shouldEvict, err := r.shouldEvict(ctx, machine)
-	if err != nil {
-		return nil, err
+func (r *Reconciler) deleteMachine(
+	ctx context.Context,
+	prov cloudprovidertypes.Provider,
+	providerName providerconfigtypes.CloudProvider,
+	machine *clusterv1alpha1.Machine,
+	skipEviction bool,
+) (*reconcile.Result, error) {
+	var (
+		shouldEvict bool
+		err         error
+	)
+
+	if !skipEviction {
+		shouldEvict, err = r.shouldEvict(ctx, machine)
+		if err != nil {
+			return nil, err
+		}
 	}
+
 	shouldCleanUpVolumes, err := r.shouldCleanupVolumes(ctx, machine, providerName)
 	if err != nil {
 		return nil, err
@@ -667,7 +686,6 @@ func (r *Reconciler) deleteCloudProviderInstance(ctx context.Context, prov cloud
 		message := fmt.Sprintf("%v. Please manually delete %s finalizer from the machine object.", err, FinalizerDeleteInstance)
 		return nil, r.updateMachineErrorIfTerminalError(machine, common.DeleteMachineError, message, err, "failed to delete machine at cloud provider")
 	}
-
 	if !completelyGone {
 		// As the instance is not completely gone yet, we need to recheck in a few seconds.
 		return &reconcile.Result{RequeueAfter: deletionRetryWaitPeriod}, nil
@@ -1036,14 +1054,6 @@ func (r *Reconciler) ensureNodeLabelsAnnotationsAndTaints(ctx context.Context, n
 		modifiers = append(modifiers, f(AnnotationAutoscalerIdentifier, autoscalerAnnotationValue))
 	}
 
-	taintExists := func(node *corev1.Node, taint corev1.Taint) bool {
-		for _, t := range node.Spec.Taints {
-			if t.MatchTaint(&taint) {
-				return true
-			}
-		}
-		return false
-	}
 	for _, t := range machine.Spec.Taints {
 		if !taintExists(node, t) {
 			f := func(t corev1.Taint) func(*corev1.Node) {
@@ -1166,6 +1176,15 @@ func findNodeByProviderID(instance instance.Instance, provider providerconfigtyp
 	return nil
 }
 
+func taintExists(node *corev1.Node, taint corev1.Taint) bool {
+	for _, t := range node.Spec.Taints {
+		if t.MatchTaint(&taint) {
+			return true
+		}
+	}
+	return false
+}
+
 func (r *Reconciler) ReadinessChecks(ctx context.Context) map[string]healthcheck.Check {
 	return map[string]healthcheck.Check{
 		"valid-info-kubeconfig": func() error {
@@ -1225,3 +1244,45 @@ func (r *Reconciler) updateNode(ctx context.Context, node *corev1.Node, modifier
 		return r.client.Update(ctx, node)
 	})
 }
+
+// handleNodeFailuresWithExternalCCM reacts to node status discovery of CCM's node lifecycle controller.
+// If an instance at cloud provider is not found then it waits till CCM deletes node objects, that allows:
+//   - create a new instance at cloud provider
+//   - initialize a new node object - the object should not be reused between instance creation
+//     for example, instance foo that got deleted and recreated should initialize a completely new node object
+//     instead of reusing the old one as it can cause problems to update node's metadata, like IP address.
+//
+// If node is shut-down it allows MC to react accordingly to specific cloud provider requirements, those are:
+//   - wait for node to become online again or
+//   - delete a machine which cannot be recovered
+func (r *Reconciler) handleNodeFailuresWithExternalCCM(
+	ctx context.Context,
+	prov cloudprovidertypes.Provider,
+	provConfig *providerconfigtypes.Config,
+	node *corev1.Node,
+	machine *clusterv1alpha1.Machine,
+) (*reconcile.Result, error) {
+	taintShutdown := corev1.Taint{
+		Key:    ccmapi.TaintNodeShutdown,
+		Effect: corev1.TaintEffectNoSchedule,
+	}
+
+	_, err := prov.Get(ctx, machine, r.providerData)
+	if err != nil {
+		if cloudprovidererrors.IsNotFound(err) {
+			klog.V(0).Infof("The node %q does not have corresponding instance, waiting for CCM to delete it", node.Name)
+			return &reconcile.Result{RequeueAfter: deletionRetryWaitPeriod}, nil
+		}
+		return nil, err
+	} else if taintExists(node, taintShutdown) {
+		switch provConfig.CloudProvider {
+		case providerconfigtypes.CloudProviderKubeVirt:
+			klog.V(0).Infof("Deleting a shut-down machine %q that cannot recover", machine.Name)
+			skipEviction := true
+			return r.deleteMachine(ctx, prov, providerconfigtypes.CloudProviderKubeVirt, machine, skipEviction)
+		}
+	}
+
+	klog.V(4).Infof("Waiting for a node to become %q", corev1.NodeReady)
+	return &reconcile.Result{RequeueAfter: deletionRetryWaitPeriod}, err
+}


Test patch:
diff --git a/pkg/cloudprovider/provider/anexia/helper_test.go b/pkg/cloudprovider/provider/anexia/helper_test.go
index 38c3a37e..c6256fa2 100644
--- a/pkg/cloudprovider/provider/anexia/helper_test.go
+++ b/pkg/cloudprovider/provider/anexia/helper_test.go
@@ -20,15 +20,27 @@ import (
 	"encoding/json"
 	"testing"
 
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+
 	"github.com/gophercloud/gophercloud/testhelper"
 
 	"github.com/kubermatic/machine-controller/pkg/apis/cluster/v1alpha1"
+	clusterv1alpha1 "github.com/kubermatic/machine-controller/pkg/apis/cluster/v1alpha1"
 	anxtypes "github.com/kubermatic/machine-controller/pkg/cloudprovider/provider/anexia/types"
+	cloudprovidertypes "github.com/kubermatic/machine-controller/pkg/cloudprovider/types"
 	"github.com/kubermatic/machine-controller/pkg/providerconfig/types"
+	providerconfigtypes "github.com/kubermatic/machine-controller/pkg/providerconfig/types"
 
 	"k8s.io/apimachinery/pkg/runtime"
 )
 
+type jsonObject = map[string]interface{}
+
+type ProvisionVMTestCase struct {
+	ReconcileContext reconcileContext
+	AssertJSONBody   func(jsonBody jsonObject)
+}
+
 type ConfigTestCase struct {
 	Config anxtypes.RawConfig
 	Error  error
@@ -67,3 +79,78 @@ func newConfigVarString(str string) types.ConfigVarString {
 		Value: str,
 	}
 }
+
+// this generates a full config and allows hooking into it to e.g. remove a value.
+func hookableConfig(hook func(*anxtypes.RawConfig)) anxtypes.RawConfig {
+	config := anxtypes.RawConfig{
+		CPUs: 1,
+
+		Memory: 2,
+
+		Disks: []anxtypes.RawDisk{
+			{Size: 5, PerformanceType: newConfigVarString("ENT6")},
+		},
+
+		Token:      newConfigVarString("test-token"),
+		VlanID:     newConfigVarString("test-vlan"),
+		LocationID: newConfigVarString("test-location"),
+		TemplateID: newConfigVarString("test-template-id"),
+	}
+
+	if hook != nil {
+		hook(&config)
+	}
+
+	return config
+}
+
+// this generates a full reconcileContext with some default values and allows hooking into it to e.g. remove/overwrite a value.
+func hookableReconcileContext(locationID string, templateID string, hook func(*reconcileContext)) reconcileContext {
+	context := reconcileContext{
+		Machine: &v1alpha1.Machine{
+			ObjectMeta: metav1.ObjectMeta{Name: "TestMachine"},
+		},
+		Status:   &anxtypes.ProviderStatus{},
+		UserData: "",
+		Config: resolvedConfig{
+			VlanID:     "VLAN-ID",
+			LocationID: locationID,
+			TemplateID: templateID,
+			Disks: []resolvedDisk{
+				{
+					RawDisk: anxtypes.RawDisk{
+						Size: 5,
+					},
+				},
+			},
+			RawConfig: anxtypes.RawConfig{
+				CPUs:   5,
+				Memory: 5,
+			},
+		},
+		ProviderData: &cloudprovidertypes.ProviderData{
+			Update: func(m *clusterv1alpha1.Machine, mods ...cloudprovidertypes.MachineModifier) error {
+				return nil
+			},
+		},
+		ProviderConfig: &providerconfigtypes.Config{
+			Network: &providerconfigtypes.NetworkConfig{
+				DNS: providerconfigtypes.DNSConfig{
+					Servers: []string{
+						"1.1.1.1",
+						"",
+						"192.168.0.1",
+						"192.168.0.2",
+						"192.168.0.3",
+					},
+				},
+			},
+		},
+	}
+
+	if hook != nil {
+		hook(&context)
+	}
+
+	return context
+}
diff --git a/pkg/cloudprovider/provider/anexia/provider_test.go b/pkg/cloudprovider/provider/anexia/provider_test.go
index a4aade62..377d5147 100644
--- a/pkg/cloudprovider/provider/anexia/provider_test.go
+++ b/pkg/cloudprovider/provider/anexia/provider_test.go
@@ -20,6 +20,7 @@ import (
 	"context"
 	"encoding/json"
 	"errors"
+	"fmt"
 	"net/http"
 	"strings"
 	"testing"
@@ -34,7 +35,6 @@ import (
 	"go.anx.io/go-anxcloud/pkg/vsphere/provisioning/vm"
 
 	"github.com/kubermatic/machine-controller/pkg/apis/cluster/v1alpha1"
-	clusterv1alpha1 "github.com/kubermatic/machine-controller/pkg/apis/cluster/v1alpha1"
 	anxtypes "github.com/kubermatic/machine-controller/pkg/cloudprovider/provider/anexia/types"
 	cloudprovidertypes "github.com/kubermatic/machine-controller/pkg/cloudprovider/types"
 	providerconfigtypes "github.com/kubermatic/machine-controller/pkg/providerconfig/types"
@@ -57,6 +57,7 @@ func TestAnexiaProvider(t *testing.T) {
 	a.FakeExisting(&vspherev1.Template{Identifier: "TEMPLATE-ID-OLD-BUILD", Name: testTemplateName, Build: "b01"})
 	a.FakeExisting(&vspherev1.Template{Identifier: "TEMPLATE-ID", Name: testTemplateName, Build: "b02"})
 	a.FakeExisting(&vspherev1.Template{Identifier: "WRONG-TEMPLATE-NAME", Name: "Wrong Template Name", Build: "b02"})
+	a.FakeExisting(&vspherev1.Template{Identifier: "TEMPLATE-ID-NO-NETWORK-CONFIG", Name: "no-network-config", Build: "b03"})
 
 	t.Cleanup(func() {
 		testhelper.TeardownHTTP()
@@ -65,6 +66,61 @@ func TestAnexiaProvider(t *testing.T) {
 
 	t.Run("Test provision VM", func(t *testing.T) {
 		t.Parallel()
+
+		testCases := []ProvisionVMTestCase{
+			{
+				// Provision a generic VM with some custom dns entries
+				ReconcileContext: hookableReconcileContext("LOCATION-ID", "TEMPLATE-ID", func(rc *reconcileContext) {
+					rc.ProviderConfig = &providerconfigtypes.Config{
+						Network: &providerconfigtypes.NetworkConfig{
+							DNS: providerconfigtypes.DNSConfig{
+								Servers: []string{
+									"1.1.1.1",
+									"",
+									"192.168.0.1",
+									"192.168.0.2",
+									"192.168.0.3",
+								},
+							},
+						},
+					}
+				}),
+				AssertJSONBody: func(jsonBody jsonObject) {
+					testhelper.AssertEquals(t, jsonBody["cpu_performance_type"], "performance")
+					testhelper.AssertEquals(t, jsonBody["hostname"], "TestMachine")
+					testhelper.AssertEquals(t, jsonBody["memory_mb"], json.Number("5"))
+
+					testhelper.AssertEquals(t, jsonBody["dns1"], "1.1.1.1")
+					_, exists := jsonBody["dns2"]
+					testhelper.AssertEquals(t, exists, false)
+					testhelper.AssertEquals(t, jsonBody["dns3"], "192.168.0.1")
+					testhelper.AssertEquals(t, jsonBody["dns4"], "192.168.0.2")
+
+					networkArray := jsonBody["network"].([]interface{})
+					networkObject := networkArray[0].(jsonObject)
+					testhelper.AssertEquals(t, networkObject["vlan"], "VLAN-ID")
+					testhelper.AssertEquals(t, networkObject["nic_type"], "vmxnet3")
+					testhelper.AssertEquals(t, networkObject["ips"].([]interface{})[0], "8.8.8.8")
+				},
+			},
+			{
+				// Provision a VM without any ProviderConfig
+				ReconcileContext: hookableReconcileContext("LOCATION-ID", "TEMPLATE-ID-NO-NETWORK-CONFIG", func(rc *reconcileContext) {
+					rc.ProviderConfig = &providerconfigtypes.Config{}
+				}),
+				AssertJSONBody: func(jsonBody jsonObject) {
+					_, exists := jsonBody["dns1"]
+					testhelper.AssertEquals(t, exists, false)
+					_, exists = jsonBody["dns2"]
+					testhelper.AssertEquals(t, exists, false)
+					_, exists = jsonBody["dns3"]
+					testhelper.AssertEquals(t, exists, false)
+					_, exists = jsonBody["dns4"]
+					testhelper.AssertEquals(t, exists, false)
+				},
+			},
+		}
+
 		testhelper.Mux.HandleFunc("/api/ipam/v1/address/reserve/ip/count.json", func(writer http.ResponseWriter, request *http.Request) {
 			err := json.NewEncoder(writer).Encode(address.ReserveRandomSummary{
 				Data: []address.ReservedIP{
@@ -77,108 +133,46 @@ func TestAnexiaProvider(t *testing.T) {
 			testhelper.AssertNoErr(t, err)
 		})
 
-		testhelper.Mux.HandleFunc("/api/vsphere/v1/provisioning/vm.json/LOCATION-ID/templates/TEMPLATE-ID", func(writer http.ResponseWriter, request *http.Request) {
-			testhelper.TestMethod(t, request, http.MethodPost)
-			type jsonObject = map[string]interface{}
-			expectedJSON := map[string]interface{}{
-				"cpu_performance_type": "performance",
-				"hostname":             "TestMachine",
-				"memory_mb":            json.Number("5"),
-				"network": []jsonObject{
-					{
-						"vlan":     "VLAN-ID",
-						"nic_type": "vmxnet3",
-						"ips":      []interface{}{"8.8.8.8"},
-					},
-				},
-			}
-			var jsonBody jsonObject
-			decoder := json.NewDecoder(request.Body)
-			decoder.UseNumber()
-			testhelper.AssertNoErr(t, decoder.Decode(&jsonBody))
-			testhelper.AssertEquals(t, expectedJSON["cpu_performance_type"], jsonBody["cpu_performance_type"])
-			testhelper.AssertEquals(t, expectedJSON["hostname"], jsonBody["hostname"])
-			testhelper.AssertEquals(t, expectedJSON["memory_mb"], jsonBody["memory_mb"])
-
-			testhelper.AssertEquals(t, jsonBody["dns1"], "1.1.1.1")
-			testhelper.AssertEquals(t, jsonBody["dns2"], nil)
-			testhelper.AssertEquals(t, jsonBody["dns3"], "192.168.0.1")
-			testhelper.AssertEquals(t, jsonBody["dns4"], "192.168.0.2")
-			testhelper.AssertEquals(t, expectedJSON["count"], jsonBody["count"])
-
-			expectedNetwork := expectedJSON["network"].([]jsonObject)[0]
-			bodyNetwork := jsonBody["network"].([]interface{})[0].(jsonObject)
-			testhelper.AssertEquals(t, expectedNetwork["vlan"], bodyNetwork["vlan"])
-			testhelper.AssertEquals(t, expectedNetwork["nic_type"], bodyNetwork["nic_type"])
-			testhelper.AssertEquals(t, expectedNetwork["ips"].([]interface{})[0], bodyNetwork["ips"].([]interface{})[0])
-
-			err := json.NewEncoder(writer).Encode(vm.ProvisioningResponse{
-				Progress:   100,
-				Errors:     nil,
-				Identifier: "TEST-IDENTIFIER",
-				Queued:     false,
+		for _, testCase := range testCases {
+			templateID := testCase.ReconcileContext.Config.TemplateID
+			locationID := testCase.ReconcileContext.Config.LocationID
+
+			testhelper.Mux.HandleFunc(fmt.Sprintf("/api/vsphere/v1/provisioning/vm.json/%s/templates/%s", locationID, templateID), func(writer http.ResponseWriter, request *http.Request) {
+				testhelper.TestMethod(t, request, http.MethodPost)
+				var jsonBody jsonObject
+				decoder := json.NewDecoder(request.Body)
+				decoder.UseNumber()
+				testhelper.AssertNoErr(t, decoder.Decode(&jsonBody))
+
+				testCase.AssertJSONBody(jsonBody)
+
+				err := json.NewEncoder(writer).Encode(vm.ProvisioningResponse{
+					Progress:   100,
+					Errors:     nil,
+					Identifier: templateID,
+					Queued:     false,
+				})
+				testhelper.AssertNoErr(t, err)
 			})
-			testhelper.AssertNoErr(t, err)
-		})
 
-		testhelper.Mux.HandleFunc("/api/vsphere/v1/provisioning/progress.json/TEST-IDENTIFIER", func(writer http.ResponseWriter, request *http.Request) {
-			testhelper.TestMethod(t, request, http.MethodGet)
+			testhelper.Mux.HandleFunc(fmt.Sprintf("/api/vsphere/v1/provisioning/progress.json/%s", templateID), func(writer http.ResponseWriter, request *http.Request) {
+				testhelper.TestMethod(t, request, http.MethodGet)
 
-			err := json.NewEncoder(writer).Encode(progress.Progress{
-				TaskIdentifier: "TEST-IDENTIFIER",
-				Queued:         false,
-				Progress:       100,
-				VMIdentifier:   "VM-IDENTIFIER",
-				Errors:         nil,
+				err := json.NewEncoder(writer).Encode(progress.Progress{
+					TaskIdentifier: templateID,
+					Queued:         false,
+					Progress:       100,
+					VMIdentifier:   "VM-IDENTIFIER",
+					Errors:         nil,
+				})
+				testhelper.AssertNoErr(t, err)
 			})
-			testhelper.AssertNoErr(t, err)
-		})
 
-		providerStatus := anxtypes.ProviderStatus{}
-		ctx := createReconcileContext(context.Background(), reconcileContext{
-			Machine: &v1alpha1.Machine{
-				ObjectMeta: metav1.ObjectMeta{Name: "TestMachine"},
-			},
-			Status:   &providerStatus,
-			UserData: "",
-			Config: resolvedConfig{
-				VlanID:     "VLAN-ID",
-				LocationID: "LOCATION-ID",
-				TemplateID: "TEMPLATE-ID",
-				Disks: []resolvedDisk{
-					{
-						RawDisk: anxtypes.RawDisk{
-							Size: 5,
-						},
-					},
-				},
-				RawConfig: anxtypes.RawConfig{
-					CPUs:   5,
-					Memory: 5,
-				},
-			},
-			ProviderData: &cloudprovidertypes.ProviderData{
-				Update: func(m *clusterv1alpha1.Machine, mods ...cloudprovidertypes.MachineModifier) error {
-					return nil
-				},
-			},
-			ProviderConfig: &providerconfigtypes.Config{
-				Network: &providerconfigtypes.NetworkConfig{
-					DNS: providerconfigtypes.DNSConfig{
-						Servers: []string{
-							"1.1.1.1",
-							"",
-							"192.168.0.1",
-							"192.168.0.2",
-							"192.168.0.3",
-						},
-					},
-				},
-			},
-		})
+			ctx := createReconcileContext(context.Background(), testCase.ReconcileContext)
 
-		err := provisionVM(ctx, client)
-		testhelper.AssertNoErr(t, err)
+			err := provisionVM(ctx, client)
+			testhelper.AssertNoErr(t, err)
+		}
 	})
 
 	t.Run("Test resolve template", func(t *testing.T) {
@@ -289,30 +283,6 @@ func TestAnexiaProvider(t *testing.T) {
 	})
 }
 
-// this generates a full config and allows hooking into it to e.g. remove a value.
-func hookableConfig(hook func(*anxtypes.RawConfig)) anxtypes.RawConfig {
-	config := anxtypes.RawConfig{
-		CPUs: 1,
-
-		Memory: 2,
-
-		Disks: []anxtypes.RawDisk{
-			{Size: 5, PerformanceType: newConfigVarString("ENT6")},
-		},
-
-		Token:      newConfigVarString("test-token"),
-		VlanID:     newConfigVarString("test-vlan"),
-		LocationID: newConfigVarString("test-location"),
-		TemplateID: newConfigVarString("test-template-id"),
-	}
-
-	if hook != nil {
-		hook(&config)
-	}
-
-	return config
-}
-
 func TestValidate(t *testing.T) {
 	t.Parallel()
 
diff --git a/test/e2e/provisioning/testdata/machinedeployment-kubevirt.yaml b/test/e2e/provisioning/testdata/machinedeployment-kubevirt.yaml
index 907e295e..dc6f691c 100644
--- a/test/e2e/provisioning/testdata/machinedeployment-kubevirt.yaml
+++ b/test/e2e/provisioning/testdata/machinedeployment-kubevirt.yaml
@@ -36,7 +36,7 @@ spec:
                 primaryDisk:
                   osImage: http://image-repo.kube-system.svc/images/<< KUBEVIRT_OS_IMAGE >>.img
                   size: "25Gi"
-                  storageClassName: px-csi-db
+                  storageClassName: rook-ceph-block
               dnsPolicy: "None"
               dnsConfig:
                 nameservers: