Repository: kubernetes-sigs/cluster-api
Pull request #2945 (2020-04-21 17:05:43)

🏃[e2e] Add MachineDeployment upgrade test
**What this PR does / why we need it**:
This PR adds MachineDeployment rolling upgrade test by modifying kubernetes version.

/area testing
/assign @fabriziopandini 

k8s-ci-robot: [APPROVALNOTIFIER] This PR is **NOT APPROVED**

This pull-request has been approved by: *<a href="https://github.com/kubernetes-sigs/cluster-api/pull/2945#" title="Author self-approved">sedefsavas</a>*
To complete the [pull request process](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process), please assign **ncdc**
You can assign the PR to them by writing `/assign @ncdc` in a comment when ready.

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=kubernetes-sigs%2Fcluster-api).

<details open>
Needs approval from an approver in each of these files:

- **[OWNERS](https://github.com/kubernetes-sigs/cluster-api/blob/master/OWNERS)**

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":["ncdc"]} -->
k8s-ci-robot: @sedefsavas: The following test **failed**, say `/retest` to rerun all failed tests:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
pull-cluster-api-apidiff | e6afab0a6765e05e30e308d50ac5eb1c10d9b5f0 | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-apidiff/1252644826256510977) | `/test pull-cluster-api-apidiff`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cluster-api&pr=2945). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3sedefsavas). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cluster-api/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @sedefsavas: The following tests **failed**, say `/retest` to rerun all failed tests:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
pull-cluster-api-apidiff | e6afab0a6765e05e30e308d50ac5eb1c10d9b5f0 | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-apidiff/1252644826256510977) | `/test pull-cluster-api-apidiff`
pull-cluster-api-make | e6afab0a6765e05e30e308d50ac5eb1c10d9b5f0 | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-make/1252644826252316675) | `/test pull-cluster-api-make`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cluster-api&pr=2945). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3sedefsavas). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cluster-api/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @sedefsavas: The following tests **failed**, say `/retest` to rerun all failed tests:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
pull-cluster-api-make | e6afab0a6765e05e30e308d50ac5eb1c10d9b5f0 | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-make/1252644826252316675) | `/test pull-cluster-api-make`
pull-cluster-api-apidiff | 25497453414c109070d354021791d96f5a2a6ecb | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-apidiff/1252646713777197057) | `/test pull-cluster-api-apidiff`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cluster-api&pr=2945). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3sedefsavas). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cluster-api/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
sedefsavas: This test do not succeed on vsphere https://github.com/kubernetes-sigs/cluster-api/pull/2950 because it relies on "cluster.x-k8s.io/deployment-name" label that is added by CAPI to MachineDeployments.
This label is only added when there is no other labels in the MachineDeployment. CAPD MachineDeployments have this label whereas CAPV deployments do not. If this is the expected behaviour (this label is not guaranteed to be in MDs) then I will update this and will not use that label to get the machines that belong to a Machine Deployment. 

Any idea on this @vincepri @fabriziopandini ?

detiber: > This label is only added when there is no other labels in the MachineDeployment. CAPD MachineDeployments have this label whereas CAPV deployments do not. If this is the expected behaviour (this label is not guaranteed to be in MDs) then I will update this and will not use that label to get the machines that belong to a Machine Deployment.

That is indeed the correct behavior, the label is only added if no labels are specified. It does this as a way to provide a default label selector for Machines when a user does not provide their own. Prior to v1alpha3 users were required to define a label selector for Machines and would get an error if they failed to do so.
sedefsavas: I can use some feedback on the test scenarios in [ test matrix](https://docs.google.com/spreadsheets/d/1uB3DyacOLctRjbI6ov7mVoRb6PnM4ktTABxBygt5sKI/edit#gid=0)  below:
  - **MachineDeployment/rolling update/Change the infrastructure template reference:** This is not possible for CAPD as there is no field to change in its template. 
  - **MachineDeployment/rolling update/Change the bootstrap template reference**: Same as above.
  - **MachineDeployment/Deletes 1 machine with a MHC**: How to trigger MHC to delete a machine?

ncdc: For the first 2 points: Can you create a template with a new name and update the MD to point at the new template?

For the MHC question: you need the node to change its status to NotReady or Unknown, or I think have the Machine's failureReason/failureMessage set. You may be able to achieve this by deleting the underlying container for the node (CAPD) and waiting a bit, or by setting failureReason/Message directly, perhaps?
detiber: > MachineDeployment/Deletes 1 machine with a MHC: How to trigger MHC to delete a machine?

You should be able to define a health check against a specific "NodeCondition" that we test for: https://godoc.org/k8s.io/api/core/v1#NodeCondition

Getting it to trigger should just be a matter of injecting the specific "NodeCondition" that we configured in the health check against the Node resource backing the Machine we want to trigger deletion for.
k8s-ci-robot: @sedefsavas: The following test **failed**, say `/retest` to rerun all failed tests:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
pull-cluster-api-apidiff | 8edab7e599e9499e61c2ba2879b450e206c13cb0 | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-apidiff/1253732944560787456) | `/test pull-cluster-api-apidiff`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cluster-api&pr=2945). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3sedefsavas). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cluster-api/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @sedefsavas: The following tests **failed**, say `/retest` to rerun all failed tests:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
pull-cluster-api-apidiff | 8edab7e599e9499e61c2ba2879b450e206c13cb0 | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-apidiff/1253732944560787456) | `/test pull-cluster-api-apidiff`
pull-cluster-api-make | 8edab7e599e9499e61c2ba2879b450e206c13cb0 | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-make/1253732944556593152) | `/test pull-cluster-api-make`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cluster-api&pr=2945). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3sedefsavas). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cluster-api/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @sedefsavas: The following tests **failed**, say `/retest` to rerun all failed tests:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
pull-cluster-api-make | 8edab7e599e9499e61c2ba2879b450e206c13cb0 | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-make/1253732944556593152) | `/test pull-cluster-api-make`
pull-cluster-api-apidiff | ba9536fe7d82c00cf5ffa7d609adc22aaabeda2f | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-apidiff/1253735586926170121) | `/test pull-cluster-api-apidiff`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cluster-api&pr=2945). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3sedefsavas). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cluster-api/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
sedefsavas: I will address rest of the tests in a separate PR for the sake of making it on time for 0.3.4 release.
Any other comments for this test? 
cc @fabriziopandini @detiber 
k8s-ci-robot: @sedefsavas: The following test **failed**, say `/retest` to rerun all failed tests:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
pull-cluster-api-apidiff | 7deeb7987d2ca831e5d4a2a02f030a07e6be898b | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-apidiff/1255027878475599873) | `/test pull-cluster-api-apidiff`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cluster-api&pr=2945). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3sedefsavas). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cluster-api/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @sedefsavas: The following tests **failed**, say `/retest` to rerun all failed tests:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
pull-cluster-api-apidiff | 7deeb7987d2ca831e5d4a2a02f030a07e6be898b | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-apidiff/1255027878475599873) | `/test pull-cluster-api-apidiff`
pull-cluster-api-e2e | 7deeb7987d2ca831e5d4a2a02f030a07e6be898b | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-e2e/1255027878471405569) | `/test pull-cluster-api-e2e`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cluster-api&pr=2945). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3sedefsavas). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cluster-api/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @sedefsavas: The following tests **failed**, say `/retest` to rerun all failed tests:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
pull-cluster-api-apidiff | 7deeb7987d2ca831e5d4a2a02f030a07e6be898b | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-apidiff/1255027878475599873) | `/test pull-cluster-api-apidiff`
pull-cluster-api-e2e | 7deeb7987d2ca831e5d4a2a02f030a07e6be898b | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-e2e/1255027878471405569) | `/test pull-cluster-api-e2e`
pull-cluster-api-capd-e2e | 7deeb7987d2ca831e5d4a2a02f030a07e6be898b | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-capd-e2e/1255027878479794176) | `/test pull-cluster-api-capd-e2e`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cluster-api&pr=2945). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3sedefsavas). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cluster-api/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @sedefsavas: The following tests **failed**, say `/retest` to rerun all failed tests:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
pull-cluster-api-e2e | 7deeb7987d2ca831e5d4a2a02f030a07e6be898b | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-e2e/1255027878471405569) | `/test pull-cluster-api-e2e`
pull-cluster-api-capd-e2e | 7deeb7987d2ca831e5d4a2a02f030a07e6be898b | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-capd-e2e/1255027878479794176) | `/test pull-cluster-api-capd-e2e`
pull-cluster-api-apidiff | 13a7b36724816535a3b83e41aac9312b3dcead24 | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-apidiff/1255039832338468867) | `/test pull-cluster-api-apidiff`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cluster-api&pr=2945). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3sedefsavas). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cluster-api/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @sedefsavas: The following tests **failed**, say `/retest` to rerun all failed tests:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
pull-cluster-api-capd-e2e | 7deeb7987d2ca831e5d4a2a02f030a07e6be898b | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-capd-e2e/1255027878479794176) | `/test pull-cluster-api-capd-e2e`
pull-cluster-api-apidiff | 13a7b36724816535a3b83e41aac9312b3dcead24 | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-apidiff/1255039832338468867) | `/test pull-cluster-api-apidiff`
pull-cluster-api-e2e | 13a7b36724816535a3b83e41aac9312b3dcead24 | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-e2e/1255039832338468868) | `/test pull-cluster-api-e2e`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cluster-api&pr=2945). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3sedefsavas). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cluster-api/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
sedefsavas: /retest pull-cluster-api-e2e
k8s-ci-robot: @sedefsavas: The `/retest` command does not accept any targets.
The following commands are available to trigger jobs:
* `/test pull-cluster-api-build`
* `/test pull-cluster-api-make`
* `/test pull-cluster-api-apidiff`
* `/test pull-cluster-api-verify`
* `/test pull-cluster-api-test`
* `/test pull-cluster-api-integration`
* `/test pull-cluster-api-capd-e2e`
* `/test pull-cluster-api-e2e`

Use `/test all` to run all jobs.

<details>

In response to [this](https://github.com/kubernetes-sigs/cluster-api/pull/2945#issuecomment-620724607):

>/retest pull-cluster-api-e2e


Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
</details>
sedefsavas: /test pull-cluster-api-test
k8s-ci-robot: @sedefsavas: The following tests **failed**, say `/retest` to rerun all failed tests:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
pull-cluster-api-e2e | 13a7b36724816535a3b83e41aac9312b3dcead24 | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-e2e/1255039832338468868) | `/test pull-cluster-api-e2e`
pull-cluster-api-apidiff | 9a5febada21fbdd7801f4e1f36a6aa70813ed1c8 | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-apidiff/1255176985932468224) | `/test pull-cluster-api-apidiff`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cluster-api&pr=2945). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3sedefsavas). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cluster-api/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
sedefsavas: This one is ready @fabriziopandini @vincepri 
k8s-ci-robot: @sedefsavas: PR needs rebase.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
</details>
k8s-ci-robot: [APPROVALNOTIFIER] This PR is **NOT APPROVED**

This pull-request has been approved by: *<a href="https://github.com/kubernetes-sigs/cluster-api/pull/2945#" title="Author self-approved">sedefsavas</a>*
To complete the [pull request process](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process), please assign **justinsb**
You can assign the PR to them by writing `/assign @justinsb` in a comment when ready.

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=kubernetes-sigs%2Fcluster-api).

<details open>
Needs approval from an approver in each of these files:

- **[OWNERS](https://github.com/kubernetes-sigs/cluster-api/blob/master/OWNERS)**

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":["justinsb"]} -->
k8s-ci-robot: @sedefsavas: The following tests **failed**, say `/retest` to rerun all failed tests:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
pull-cluster-api-apidiff | 9a5febada21fbdd7801f4e1f36a6aa70813ed1c8 | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-apidiff/1255176985932468224) | `/test pull-cluster-api-apidiff`
pull-cluster-api-make | c61bec8c0b91ce34713f7fc8fa0765a2ae03783d | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-make/1255902699988193281) | `/test pull-cluster-api-make`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cluster-api&pr=2945). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3sedefsavas). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cluster-api/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @sedefsavas: The following tests **failed**, say `/retest` to rerun all failed tests:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
pull-cluster-api-apidiff | 9a5febada21fbdd7801f4e1f36a6aa70813ed1c8 | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-apidiff/1255176985932468224) | `/test pull-cluster-api-apidiff`
pull-cluster-api-make | c61bec8c0b91ce34713f7fc8fa0765a2ae03783d | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-make/1255902699988193281) | `/test pull-cluster-api-make`
pull-cluster-api-e2e | c61bec8c0b91ce34713f7fc8fa0765a2ae03783d | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-e2e/1255902699992387584) | `/test pull-cluster-api-e2e`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cluster-api&pr=2945). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3sedefsavas). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cluster-api/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @sedefsavas: The following tests **failed**, say `/retest` to rerun all failed tests:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
pull-cluster-api-apidiff | 9a5febada21fbdd7801f4e1f36a6aa70813ed1c8 | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-apidiff/1255176985932468224) | `/test pull-cluster-api-apidiff`
pull-cluster-api-make | c61bec8c0b91ce34713f7fc8fa0765a2ae03783d | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-make/1255902699988193281) | `/test pull-cluster-api-make`
pull-cluster-api-e2e | c61bec8c0b91ce34713f7fc8fa0765a2ae03783d | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-e2e/1255902699992387584) | `/test pull-cluster-api-e2e`
pull-cluster-api-capd-e2e | c61bec8c0b91ce34713f7fc8fa0765a2ae03783d | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-capd-e2e/1255902699988193282) | `/test pull-cluster-api-capd-e2e`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cluster-api&pr=2945). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3sedefsavas). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cluster-api/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @sedefsavas: The following tests **failed**, say `/retest` to rerun all failed tests:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
pull-cluster-api-apidiff | 9a5febada21fbdd7801f4e1f36a6aa70813ed1c8 | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-apidiff/1255176985932468224) | `/test pull-cluster-api-apidiff`
pull-cluster-api-make | c61bec8c0b91ce34713f7fc8fa0765a2ae03783d | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-make/1255902699988193281) | `/test pull-cluster-api-make`
pull-cluster-api-e2e | c61bec8c0b91ce34713f7fc8fa0765a2ae03783d | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-e2e/1255902699992387584) | `/test pull-cluster-api-e2e`
pull-cluster-api-capd-e2e | c61bec8c0b91ce34713f7fc8fa0765a2ae03783d | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-capd-e2e/1255902699988193282) | `/test pull-cluster-api-capd-e2e`
pull-cluster-api-integration | c61bec8c0b91ce34713f7fc8fa0765a2ae03783d | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-integration/1255902699988193283) | `/test pull-cluster-api-integration`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cluster-api&pr=2945). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3sedefsavas). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cluster-api/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @sedefsavas: The following tests **failed**, say `/retest` to rerun all failed tests:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
pull-cluster-api-apidiff | 9a5febada21fbdd7801f4e1f36a6aa70813ed1c8 | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-apidiff/1255176985932468224) | `/test pull-cluster-api-apidiff`
pull-cluster-api-make | c61bec8c0b91ce34713f7fc8fa0765a2ae03783d | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-make/1255902699988193281) | `/test pull-cluster-api-make`
pull-cluster-api-e2e | c61bec8c0b91ce34713f7fc8fa0765a2ae03783d | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-e2e/1255902699992387584) | `/test pull-cluster-api-e2e`
pull-cluster-api-capd-e2e | c61bec8c0b91ce34713f7fc8fa0765a2ae03783d | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-capd-e2e/1255902699988193282) | `/test pull-cluster-api-capd-e2e`
pull-cluster-api-integration | c61bec8c0b91ce34713f7fc8fa0765a2ae03783d | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-integration/1255902699988193283) | `/test pull-cluster-api-integration`
pull-cluster-api-test | c61bec8c0b91ce34713f7fc8fa0765a2ae03783d | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-test/1255902699992387585) | `/test pull-cluster-api-test`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cluster-api&pr=2945). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3sedefsavas). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cluster-api/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @sedefsavas: The following tests **failed**, say `/retest` to rerun all failed tests:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
pull-cluster-api-apidiff | 9a5febada21fbdd7801f4e1f36a6aa70813ed1c8 | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-apidiff/1255176985932468224) | `/test pull-cluster-api-apidiff`
pull-cluster-api-make | c61bec8c0b91ce34713f7fc8fa0765a2ae03783d | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-make/1255902699988193281) | `/test pull-cluster-api-make`
pull-cluster-api-e2e | c61bec8c0b91ce34713f7fc8fa0765a2ae03783d | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-e2e/1255902699992387584) | `/test pull-cluster-api-e2e`
pull-cluster-api-capd-e2e | c61bec8c0b91ce34713f7fc8fa0765a2ae03783d | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-capd-e2e/1255902699988193282) | `/test pull-cluster-api-capd-e2e`
pull-cluster-api-integration | c61bec8c0b91ce34713f7fc8fa0765a2ae03783d | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-integration/1255902699988193283) | `/test pull-cluster-api-integration`
pull-cluster-api-test | c61bec8c0b91ce34713f7fc8fa0765a2ae03783d | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-test/1255902699992387585) | `/test pull-cluster-api-test`
pull-cluster-api-build | 4b253b054fab417ba2bd6be9596d47b007c1761a | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-build/1255903454165995522) | `/test pull-cluster-api-build`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cluster-api&pr=2945). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3sedefsavas). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cluster-api/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @sedefsavas: The following tests **failed**, say `/retest` to rerun all failed tests:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
pull-cluster-api-make | c61bec8c0b91ce34713f7fc8fa0765a2ae03783d | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-make/1255902699988193281) | `/test pull-cluster-api-make`
pull-cluster-api-e2e | c61bec8c0b91ce34713f7fc8fa0765a2ae03783d | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-e2e/1255902699992387584) | `/test pull-cluster-api-e2e`
pull-cluster-api-capd-e2e | c61bec8c0b91ce34713f7fc8fa0765a2ae03783d | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-capd-e2e/1255902699988193282) | `/test pull-cluster-api-capd-e2e`
pull-cluster-api-integration | c61bec8c0b91ce34713f7fc8fa0765a2ae03783d | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-integration/1255902699988193283) | `/test pull-cluster-api-integration`
pull-cluster-api-test | c61bec8c0b91ce34713f7fc8fa0765a2ae03783d | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-test/1255902699992387585) | `/test pull-cluster-api-test`
pull-cluster-api-build | 4b253b054fab417ba2bd6be9596d47b007c1761a | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-build/1255903454165995522) | `/test pull-cluster-api-build`
pull-cluster-api-apidiff | 4b253b054fab417ba2bd6be9596d47b007c1761a | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-apidiff/1255903454165995521) | `/test pull-cluster-api-apidiff`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cluster-api&pr=2945). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3sedefsavas). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cluster-api/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @sedefsavas: The following tests **failed**, say `/retest` to rerun all failed tests:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
pull-cluster-api-make | c61bec8c0b91ce34713f7fc8fa0765a2ae03783d | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-make/1255902699988193281) | `/test pull-cluster-api-make`
pull-cluster-api-capd-e2e | c61bec8c0b91ce34713f7fc8fa0765a2ae03783d | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-capd-e2e/1255902699988193282) | `/test pull-cluster-api-capd-e2e`
pull-cluster-api-integration | c61bec8c0b91ce34713f7fc8fa0765a2ae03783d | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-integration/1255902699988193283) | `/test pull-cluster-api-integration`
pull-cluster-api-test | c61bec8c0b91ce34713f7fc8fa0765a2ae03783d | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-test/1255902699992387585) | `/test pull-cluster-api-test`
pull-cluster-api-build | 4b253b054fab417ba2bd6be9596d47b007c1761a | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-build/1255903454165995522) | `/test pull-cluster-api-build`
pull-cluster-api-apidiff | 4b253b054fab417ba2bd6be9596d47b007c1761a | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-apidiff/1255903454165995521) | `/test pull-cluster-api-apidiff`
pull-cluster-api-e2e | 4b253b054fab417ba2bd6be9596d47b007c1761a | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-e2e/1255903454165995524) | `/test pull-cluster-api-e2e`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cluster-api&pr=2945). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3sedefsavas). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cluster-api/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @sedefsavas: The following tests **failed**, say `/retest` to rerun all failed tests:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
pull-cluster-api-capd-e2e | c61bec8c0b91ce34713f7fc8fa0765a2ae03783d | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-capd-e2e/1255902699988193282) | `/test pull-cluster-api-capd-e2e`
pull-cluster-api-integration | c61bec8c0b91ce34713f7fc8fa0765a2ae03783d | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-integration/1255902699988193283) | `/test pull-cluster-api-integration`
pull-cluster-api-test | c61bec8c0b91ce34713f7fc8fa0765a2ae03783d | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-test/1255902699992387585) | `/test pull-cluster-api-test`
pull-cluster-api-build | 4b253b054fab417ba2bd6be9596d47b007c1761a | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-build/1255903454165995522) | `/test pull-cluster-api-build`
pull-cluster-api-apidiff | 4b253b054fab417ba2bd6be9596d47b007c1761a | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-apidiff/1255903454165995521) | `/test pull-cluster-api-apidiff`
pull-cluster-api-e2e | 4b253b054fab417ba2bd6be9596d47b007c1761a | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-e2e/1255903454165995524) | `/test pull-cluster-api-e2e`
pull-cluster-api-make | 4b253b054fab417ba2bd6be9596d47b007c1761a | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-make/1255903454165995520) | `/test pull-cluster-api-make`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cluster-api&pr=2945). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3sedefsavas). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cluster-api/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @sedefsavas: The following tests **failed**, say `/retest` to rerun all failed tests:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
pull-cluster-api-capd-e2e | c61bec8c0b91ce34713f7fc8fa0765a2ae03783d | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-capd-e2e/1255902699988193282) | `/test pull-cluster-api-capd-e2e`
pull-cluster-api-integration | c61bec8c0b91ce34713f7fc8fa0765a2ae03783d | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-integration/1255902699988193283) | `/test pull-cluster-api-integration`
pull-cluster-api-test | c61bec8c0b91ce34713f7fc8fa0765a2ae03783d | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-test/1255902699992387585) | `/test pull-cluster-api-test`
pull-cluster-api-build | 4b253b054fab417ba2bd6be9596d47b007c1761a | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-build/1255903454165995522) | `/test pull-cluster-api-build`
pull-cluster-api-apidiff | 4b253b054fab417ba2bd6be9596d47b007c1761a | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-apidiff/1255903454165995521) | `/test pull-cluster-api-apidiff`
pull-cluster-api-e2e | 4b253b054fab417ba2bd6be9596d47b007c1761a | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-e2e/1255903454165995524) | `/test pull-cluster-api-e2e`
pull-cluster-api-make | 4b253b054fab417ba2bd6be9596d47b007c1761a | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-make/1255903454165995520) | `/test pull-cluster-api-make`
pull-cluster-api-verify | 4b253b054fab417ba2bd6be9596d47b007c1761a | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-verify/1255903454165995523) | `/test pull-cluster-api-verify`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cluster-api&pr=2945). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3sedefsavas). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cluster-api/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @sedefsavas: The following tests **failed**, say `/retest` to rerun all failed tests:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
pull-cluster-api-capd-e2e | c61bec8c0b91ce34713f7fc8fa0765a2ae03783d | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-capd-e2e/1255902699988193282) | `/test pull-cluster-api-capd-e2e`
pull-cluster-api-integration | c61bec8c0b91ce34713f7fc8fa0765a2ae03783d | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-integration/1255902699988193283) | `/test pull-cluster-api-integration`
pull-cluster-api-build | 4b253b054fab417ba2bd6be9596d47b007c1761a | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-build/1255903454165995522) | `/test pull-cluster-api-build`
pull-cluster-api-apidiff | 4b253b054fab417ba2bd6be9596d47b007c1761a | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-apidiff/1255903454165995521) | `/test pull-cluster-api-apidiff`
pull-cluster-api-e2e | 4b253b054fab417ba2bd6be9596d47b007c1761a | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-e2e/1255903454165995524) | `/test pull-cluster-api-e2e`
pull-cluster-api-make | 4b253b054fab417ba2bd6be9596d47b007c1761a | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-make/1255903454165995520) | `/test pull-cluster-api-make`
pull-cluster-api-verify | 4b253b054fab417ba2bd6be9596d47b007c1761a | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-verify/1255903454165995523) | `/test pull-cluster-api-verify`
pull-cluster-api-test | 4b253b054fab417ba2bd6be9596d47b007c1761a | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-test/1255903454165995527) | `/test pull-cluster-api-test`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cluster-api&pr=2945). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3sedefsavas). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cluster-api/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @sedefsavas: The following tests **failed**, say `/retest` to rerun all failed tests:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
pull-cluster-api-integration | c61bec8c0b91ce34713f7fc8fa0765a2ae03783d | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-integration/1255902699988193283) | `/test pull-cluster-api-integration`
pull-cluster-api-build | 4b253b054fab417ba2bd6be9596d47b007c1761a | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-build/1255903454165995522) | `/test pull-cluster-api-build`
pull-cluster-api-apidiff | 4b253b054fab417ba2bd6be9596d47b007c1761a | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-apidiff/1255903454165995521) | `/test pull-cluster-api-apidiff`
pull-cluster-api-e2e | 4b253b054fab417ba2bd6be9596d47b007c1761a | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-e2e/1255903454165995524) | `/test pull-cluster-api-e2e`
pull-cluster-api-make | 4b253b054fab417ba2bd6be9596d47b007c1761a | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-make/1255903454165995520) | `/test pull-cluster-api-make`
pull-cluster-api-verify | 4b253b054fab417ba2bd6be9596d47b007c1761a | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-verify/1255903454165995523) | `/test pull-cluster-api-verify`
pull-cluster-api-test | 4b253b054fab417ba2bd6be9596d47b007c1761a | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-test/1255903454165995527) | `/test pull-cluster-api-test`
pull-cluster-api-capd-e2e | 4b253b054fab417ba2bd6be9596d47b007c1761a | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-capd-e2e/1255903454165995525) | `/test pull-cluster-api-capd-e2e`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cluster-api&pr=2945). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3sedefsavas). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cluster-api/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @sedefsavas: The following tests **failed**, say `/retest` to rerun all failed tests:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
pull-cluster-api-build | 4b253b054fab417ba2bd6be9596d47b007c1761a | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-build/1255903454165995522) | `/test pull-cluster-api-build`
pull-cluster-api-apidiff | 4b253b054fab417ba2bd6be9596d47b007c1761a | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-apidiff/1255903454165995521) | `/test pull-cluster-api-apidiff`
pull-cluster-api-e2e | 4b253b054fab417ba2bd6be9596d47b007c1761a | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-e2e/1255903454165995524) | `/test pull-cluster-api-e2e`
pull-cluster-api-make | 4b253b054fab417ba2bd6be9596d47b007c1761a | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-make/1255903454165995520) | `/test pull-cluster-api-make`
pull-cluster-api-verify | 4b253b054fab417ba2bd6be9596d47b007c1761a | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-verify/1255903454165995523) | `/test pull-cluster-api-verify`
pull-cluster-api-test | 4b253b054fab417ba2bd6be9596d47b007c1761a | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-test/1255903454165995527) | `/test pull-cluster-api-test`
pull-cluster-api-capd-e2e | 4b253b054fab417ba2bd6be9596d47b007c1761a | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-capd-e2e/1255903454165995525) | `/test pull-cluster-api-capd-e2e`
pull-cluster-api-integration | 4b253b054fab417ba2bd6be9596d47b007c1761a | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-integration/1255903454165995526) | `/test pull-cluster-api-integration`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cluster-api&pr=2945). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3sedefsavas). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cluster-api/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
sedefsavas: For **MachineDeployment/Deletes 1 machine with a MHC** test: 
I couldn't change node conditions (those fields are controlled by Kubelet I think).
As a workaround, I spinned up a cluster without CNI so node "Ready" conditions are false. MHC is detecting that the nodes are not ready (ExpectedMachines=2, CurrentHealthy=0) but not initiating remediation. After I apply a CNI to that cluster, it becomes ExpectedMachines=2, CurrentHealthy=0. Any idea why MHC does not kick in?
@detiber?
k8s-ci-robot: @sedefsavas: The following tests **failed**, say `/retest` to rerun all failed tests:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
pull-cluster-api-build | 4b253b054fab417ba2bd6be9596d47b007c1761a | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-build/1255903454165995522) | `/test pull-cluster-api-build`
pull-cluster-api-apidiff | 4b253b054fab417ba2bd6be9596d47b007c1761a | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-apidiff/1255903454165995521) | `/test pull-cluster-api-apidiff`
pull-cluster-api-e2e | 4b253b054fab417ba2bd6be9596d47b007c1761a | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-e2e/1255903454165995524) | `/test pull-cluster-api-e2e`
pull-cluster-api-make | 4b253b054fab417ba2bd6be9596d47b007c1761a | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-make/1255903454165995520) | `/test pull-cluster-api-make`
pull-cluster-api-test | 4b253b054fab417ba2bd6be9596d47b007c1761a | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-test/1255903454165995527) | `/test pull-cluster-api-test`
pull-cluster-api-capd-e2e | 4b253b054fab417ba2bd6be9596d47b007c1761a | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-capd-e2e/1255903454165995525) | `/test pull-cluster-api-capd-e2e`
pull-cluster-api-integration | 4b253b054fab417ba2bd6be9596d47b007c1761a | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-integration/1255903454165995526) | `/test pull-cluster-api-integration`
pull-cluster-api-verify | 3fd300f9f6140ea3ebe35d3d3b59dd911ec063cf | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-verify/1256213060582903811) | `/test pull-cluster-api-verify`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cluster-api&pr=2945). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3sedefsavas). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cluster-api/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @sedefsavas: The following tests **failed**, say `/retest` to rerun all failed tests:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
pull-cluster-api-e2e | 4b253b054fab417ba2bd6be9596d47b007c1761a | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-e2e/1255903454165995524) | `/test pull-cluster-api-e2e`
pull-cluster-api-make | 4b253b054fab417ba2bd6be9596d47b007c1761a | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-make/1255903454165995520) | `/test pull-cluster-api-make`
pull-cluster-api-test | 4b253b054fab417ba2bd6be9596d47b007c1761a | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-test/1255903454165995527) | `/test pull-cluster-api-test`
pull-cluster-api-capd-e2e | 4b253b054fab417ba2bd6be9596d47b007c1761a | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-capd-e2e/1255903454165995525) | `/test pull-cluster-api-capd-e2e`
pull-cluster-api-integration | 4b253b054fab417ba2bd6be9596d47b007c1761a | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-integration/1255903454165995526) | `/test pull-cluster-api-integration`
pull-cluster-api-verify | 3fd300f9f6140ea3ebe35d3d3b59dd911ec063cf | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-verify/1256213060582903811) | `/test pull-cluster-api-verify`
pull-cluster-api-apidiff | 3fd300f9f6140ea3ebe35d3d3b59dd911ec063cf | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-apidiff/1256213060578709508) | `/test pull-cluster-api-apidiff`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cluster-api&pr=2945). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3sedefsavas). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cluster-api/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @sedefsavas: The following tests **failed**, say `/retest` to rerun all failed tests:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
pull-cluster-api-capd-e2e | 4b253b054fab417ba2bd6be9596d47b007c1761a | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-capd-e2e/1255903454165995525) | `/test pull-cluster-api-capd-e2e`
pull-cluster-api-verify | 3fd300f9f6140ea3ebe35d3d3b59dd911ec063cf | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-verify/1256213060582903811) | `/test pull-cluster-api-verify`
pull-cluster-api-apidiff | 3fd300f9f6140ea3ebe35d3d3b59dd911ec063cf | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-apidiff/1256213060578709508) | `/test pull-cluster-api-apidiff`
pull-cluster-api-e2e | 3fd300f9f6140ea3ebe35d3d3b59dd911ec063cf | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-e2e/1256213060582903808) | `/test pull-cluster-api-e2e`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cluster-api&pr=2945). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3sedefsavas). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cluster-api/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
sedefsavas: /test pull-cluster-api-e2e

k8s-ci-robot: @sedefsavas: The following tests **failed**, say `/retest` to rerun all failed tests:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
pull-cluster-api-apidiff | 3fd300f9f6140ea3ebe35d3d3b59dd911ec063cf | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-apidiff/1256213060578709508) | `/test pull-cluster-api-apidiff`
pull-cluster-api-verify | 0183e74527e99287c42b578c3885b9c8058df305 | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-verify/1257224689181265926) | `/test pull-cluster-api-verify`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cluster-api&pr=2945). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3sedefsavas). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cluster-api/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @sedefsavas: The following tests **failed**, say `/retest` to rerun all failed tests:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
pull-cluster-api-verify | 0183e74527e99287c42b578c3885b9c8058df305 | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-verify/1257224689181265926) | `/test pull-cluster-api-verify`
pull-cluster-api-apidiff | 0183e74527e99287c42b578c3885b9c8058df305 | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-apidiff/1257224689181265920) | `/test pull-cluster-api-apidiff`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cluster-api&pr=2945). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3sedefsavas). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cluster-api/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @sedefsavas: The following tests **failed**, say `/retest` to rerun all failed tests:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
pull-cluster-api-verify | 0183e74527e99287c42b578c3885b9c8058df305 | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-verify/1257224689181265926) | `/test pull-cluster-api-verify`
pull-cluster-api-apidiff | a828117ecd87c8a944ea573a4ef5405b70f4599f | [link](https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/kubernetes-sigs_cluster-api/2945/pull-cluster-api-apidiff/1257226198409285634) | `/test pull-cluster-api-apidiff`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes-sigs&repo=cluster-api&pr=2945). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3sedefsavas). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes-sigs/cluster-api/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
sedefsavas: @fabriziopandini Thanks for your comments. Addressed all but making MHC as part of the cluster template. Do you mean adding MHC after ConfigCluster() aka "clusterctl config cluster" is complete?
fabriziopandini: > Do you mean adding MHC after ConfigCluster() aka "clusterctl config cluster" is complete?

No, I mean adding the MHC into the cluster-template.yaml file. This will make the MHC to be created in all the test clusters, but I don't see problems in this

Patch:
diff --git a/bootstrap/kubeadm/controllers/kubeadmconfig_controller.go b/bootstrap/kubeadm/controllers/kubeadmconfig_controller.go
index a2f99eb47..fec09f452 100644
--- a/bootstrap/kubeadm/controllers/kubeadmconfig_controller.go
+++ b/bootstrap/kubeadm/controllers/kubeadmconfig_controller.go
@@ -42,9 +42,7 @@ import (
 	expv1 "sigs.k8s.io/cluster-api/exp/api/v1alpha3"
 	"sigs.k8s.io/cluster-api/feature"
 	"sigs.k8s.io/cluster-api/util"
-	"sigs.k8s.io/cluster-api/util/annotations"
 	"sigs.k8s.io/cluster-api/util/patch"
-	"sigs.k8s.io/cluster-api/util/predicates"
 	"sigs.k8s.io/cluster-api/util/secret"
 	ctrl "sigs.k8s.io/controller-runtime"
 	"sigs.k8s.io/controller-runtime/pkg/client"
@@ -92,19 +90,24 @@ func (r *KubeadmConfigReconciler) SetupWithManager(mgr ctrl.Manager, option cont
 
 	r.scheme = mgr.GetScheme()
 
-	b := ctrl.NewControllerManagedBy(mgr).
+	builder := ctrl.NewControllerManagedBy(mgr).
 		For(&bootstrapv1.KubeadmConfig{}).
 		WithOptions(option).
-		WithEventFilter(predicates.ResourceNotPaused(r.Log)).
 		Watches(
 			&source.Kind{Type: &clusterv1.Machine{}},
 			&handler.EnqueueRequestsFromMapFunc{
 				ToRequests: handler.ToRequestsFunc(r.MachineToBootstrapMapFunc),
 			},
+		).
+		Watches(
+			&source.Kind{Type: &clusterv1.Cluster{}},
+			&handler.EnqueueRequestsFromMapFunc{
+				ToRequests: handler.ToRequestsFunc(r.ClusterToKubeadmConfigs),
+			},
 		)
 
 	if feature.Gates.Enabled(feature.MachinePool) {
-		b = b.Watches(
+		builder = builder.Watches(
 			&source.Kind{Type: &expv1.MachinePool{}},
 			&handler.EnqueueRequestsFromMapFunc{
 				ToRequests: handler.ToRequestsFunc(r.MachinePoolToBootstrapMapFunc),
@@ -112,22 +115,10 @@ func (r *KubeadmConfigReconciler) SetupWithManager(mgr ctrl.Manager, option cont
 		)
 	}
 
-	c, err := b.Build(r)
-	if err != nil {
+	if err := builder.Complete(r); err != nil {
 		return errors.Wrap(err, "failed setting up with a controller manager")
 	}
 
-	err = c.Watch(
-		&source.Kind{Type: &clusterv1.Cluster{}},
-		&handler.EnqueueRequestsFromMapFunc{
-			ToRequests: handler.ToRequestsFunc(r.ClusterToKubeadmConfigs),
-		},
-		predicates.ClusterUnpausedAndInfrastructureReady(r.Log),
-	)
-	if err != nil {
-		return errors.Wrap(err, "failed adding Watch for Clusters to controller manager")
-	}
-
 	return nil
 }
 
@@ -177,11 +168,6 @@ func (r *KubeadmConfigReconciler) Reconcile(req ctrl.Request) (_ ctrl.Result, re
 		return ctrl.Result{}, err
 	}
 
-	if annotations.IsPaused(cluster, config) {
-		log.Info("Reconciliation is paused for this object")
-		return ctrl.Result{}, nil
-	}
-
 	scope := &Scope{
 		Logger:      log,
 		Config:      config,
diff --git a/controllers/cluster_controller.go b/controllers/cluster_controller.go
index cd4541e4e..977bcc966 100644
--- a/controllers/cluster_controller.go
+++ b/controllers/cluster_controller.go
@@ -35,9 +35,7 @@ import (
 	"sigs.k8s.io/cluster-api/controllers/metrics"
 	capierrors "sigs.k8s.io/cluster-api/errors"
 	"sigs.k8s.io/cluster-api/util"
-	"sigs.k8s.io/cluster-api/util/annotations"
 	"sigs.k8s.io/cluster-api/util/patch"
-	"sigs.k8s.io/cluster-api/util/predicates"
 	"sigs.k8s.io/cluster-api/util/secret"
 	ctrl "sigs.k8s.io/controller-runtime"
 	"sigs.k8s.io/controller-runtime/pkg/client"
@@ -79,7 +77,6 @@ func (r *ClusterReconciler) SetupWithManager(mgr ctrl.Manager, options controlle
 			&handler.EnqueueRequestsFromMapFunc{ToRequests: handler.ToRequestsFunc(r.controlPlaneMachineToCluster)},
 		).
 		WithOptions(options).
-		WithEventFilter(predicates.ResourceNotPaused(r.Log)).
 		Build(r)
 
 	if err != nil {
@@ -112,7 +109,7 @@ func (r *ClusterReconciler) Reconcile(req ctrl.Request) (_ ctrl.Result, reterr e
 	}
 
 	// Return early if the object or Cluster is paused.
-	if annotations.IsPaused(cluster, cluster) {
+	if util.IsPaused(cluster, cluster) {
 		logger.Info("Reconciliation is paused for this object")
 		return ctrl.Result{}, nil
 	}
diff --git a/controllers/cluster_controller_phases.go b/controllers/cluster_controller_phases.go
index 6f815fc52..7888f5ac7 100644
--- a/controllers/cluster_controller_phases.go
+++ b/controllers/cluster_controller_phases.go
@@ -29,7 +29,6 @@ import (
 	"sigs.k8s.io/cluster-api/controllers/external"
 	capierrors "sigs.k8s.io/cluster-api/errors"
 	"sigs.k8s.io/cluster-api/util"
-	"sigs.k8s.io/cluster-api/util/annotations"
 	utilconversion "sigs.k8s.io/cluster-api/util/conversion"
 	"sigs.k8s.io/cluster-api/util/kubeconfig"
 	"sigs.k8s.io/cluster-api/util/patch"
@@ -79,7 +78,7 @@ func (r *ClusterReconciler) reconcileExternal(ctx context.Context, cluster *clus
 	}
 
 	// if external ref is paused, return error.
-	if annotations.IsPaused(cluster, obj) {
+	if util.IsPaused(cluster, obj) {
 		logger.V(3).Info("External object referenced is paused")
 		return external.ReconcileOutput{Paused: true}, nil
 	}
diff --git a/controllers/external/tracker.go b/controllers/external/tracker.go
index 29088fdac..bbc097c60 100644
--- a/controllers/external/tracker.go
+++ b/controllers/external/tracker.go
@@ -23,7 +23,6 @@ import (
 	"github.com/pkg/errors"
 	"k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"
 	"k8s.io/apimachinery/pkg/runtime"
-	"sigs.k8s.io/cluster-api/util/predicates"
 	"sigs.k8s.io/controller-runtime/pkg/controller"
 	"sigs.k8s.io/controller-runtime/pkg/handler"
 	"sigs.k8s.io/controller-runtime/pkg/source"
@@ -56,7 +55,6 @@ func (o *ObjectTracker) Watch(log logr.Logger, obj runtime.Object, handler handl
 	err := o.Controller.Watch(
 		&source.Kind{Type: u},
 		handler,
-		predicates.ResourceNotPaused(log),
 	)
 	if err != nil {
 		o.m.Delete(obj)
diff --git a/controllers/machine_controller.go b/controllers/machine_controller.go
index ab7feca5d..2298b70f0 100644
--- a/controllers/machine_controller.go
+++ b/controllers/machine_controller.go
@@ -42,16 +42,13 @@ import (
 	capierrors "sigs.k8s.io/cluster-api/errors"
 	kubedrain "sigs.k8s.io/cluster-api/third_party/kubernetes-drain"
 	"sigs.k8s.io/cluster-api/util"
-	"sigs.k8s.io/cluster-api/util/annotations"
 	"sigs.k8s.io/cluster-api/util/patch"
-	"sigs.k8s.io/cluster-api/util/predicates"
 	ctrl "sigs.k8s.io/controller-runtime"
 	"sigs.k8s.io/controller-runtime/pkg/client"
 	"sigs.k8s.io/controller-runtime/pkg/controller"
 	"sigs.k8s.io/controller-runtime/pkg/controller/controllerutil"
 	"sigs.k8s.io/controller-runtime/pkg/handler"
 	"sigs.k8s.io/controller-runtime/pkg/reconcile"
-	"sigs.k8s.io/controller-runtime/pkg/source"
 )
 
 var (
@@ -80,30 +77,22 @@ type MachineReconciler struct {
 }
 
 func (r *MachineReconciler) SetupWithManager(mgr ctrl.Manager, options controller.Options) error {
-	clusterToMachines, err := util.ClusterToObjectsMapper(mgr.GetClient(), &clusterv1.MachineList{}, mgr.GetScheme())
-	if err != nil {
-		return err
-	}
-
 	controller, err := ctrl.NewControllerManagedBy(mgr).
 		For(&clusterv1.Machine{}).
 		WithOptions(options).
-		WithEventFilter(predicates.ResourceNotPaused(r.Log)).
 		Build(r)
+
 	if err != nil {
 		return errors.Wrap(err, "failed setting up with a controller manager")
 	}
 
-	err = controller.Watch(
-		&source.Kind{Type: &clusterv1.Cluster{}},
-		&handler.EnqueueRequestsFromMapFunc{
-			ToRequests: clusterToMachines,
-		},
-		// TODO: should this wait for Cluster.Status.InfrastructureReady similar to Infra Machine resources?
-		predicates.ClusterUnpaused(r.Log),
-	)
+	// Add a watch on clusterv1.Cluster object for paused notifications.
+	clusterToMachines, err := util.ClusterToObjectsMapper(mgr.GetClient(), &clusterv1.MachineList{}, mgr.GetScheme())
 	if err != nil {
-		return errors.Wrap(err, "failed to add Watch for Clusters to controller manager")
+		return err
+	}
+	if err := util.WatchOnClusterPaused(controller, clusterToMachines); err != nil {
+		return err
 	}
 
 	r.recorder = mgr.GetEventRecorderFor("machine-controller")
@@ -154,7 +143,7 @@ func (r *MachineReconciler) Reconcile(req ctrl.Request) (_ ctrl.Result, reterr e
 	}
 
 	// Return early if the object or Cluster is paused.
-	if annotations.IsPaused(cluster, m) {
+	if util.IsPaused(cluster, m) {
 		logger.Info("Reconciliation is paused for this object")
 		return ctrl.Result{}, nil
 	}
diff --git a/controllers/machine_controller_phases.go b/controllers/machine_controller_phases.go
index 73cf303a9..1f6df26a8 100644
--- a/controllers/machine_controller_phases.go
+++ b/controllers/machine_controller_phases.go
@@ -28,7 +28,6 @@ import (
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 	"k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"
 	"k8s.io/utils/pointer"
-	"sigs.k8s.io/cluster-api/util/annotations"
 	utilconversion "sigs.k8s.io/cluster-api/util/conversion"
 	"sigs.k8s.io/controller-runtime/pkg/controller/controllerutil"
 	"sigs.k8s.io/controller-runtime/pkg/handler"
@@ -103,7 +102,7 @@ func (r *MachineReconciler) reconcileExternal(ctx context.Context, cluster *clus
 	}
 
 	// if external ref is paused, return error.
-	if annotations.IsPaused(cluster, obj) {
+	if util.IsPaused(cluster, obj) {
 		logger.V(3).Info("External object referenced is paused")
 		return external.ReconcileOutput{Paused: true}, nil
 	}
diff --git a/controllers/machinedeployment_controller.go b/controllers/machinedeployment_controller.go
index 6404c3e80..b0ced4ecb 100644
--- a/controllers/machinedeployment_controller.go
+++ b/controllers/machinedeployment_controller.go
@@ -30,9 +30,7 @@ import (
 	"k8s.io/client-go/tools/record"
 	clusterv1 "sigs.k8s.io/cluster-api/api/v1alpha3"
 	"sigs.k8s.io/cluster-api/util"
-	"sigs.k8s.io/cluster-api/util/annotations"
 	"sigs.k8s.io/cluster-api/util/patch"
-	"sigs.k8s.io/cluster-api/util/predicates"
 	ctrl "sigs.k8s.io/controller-runtime"
 	"sigs.k8s.io/controller-runtime/pkg/client"
 	"sigs.k8s.io/controller-runtime/pkg/controller"
@@ -60,12 +58,7 @@ type MachineDeploymentReconciler struct {
 }
 
 func (r *MachineDeploymentReconciler) SetupWithManager(mgr ctrl.Manager, options controller.Options) error {
-	clusterToMachineDeployments, err := util.ClusterToObjectsMapper(mgr.GetClient(), &clusterv1.MachineDeploymentList{}, mgr.GetScheme())
-	if err != nil {
-		return err
-	}
-
-	c, err := ctrl.NewControllerManagedBy(mgr).
+	controller, err := ctrl.NewControllerManagedBy(mgr).
 		For(&clusterv1.MachineDeployment{}).
 		Owns(&clusterv1.MachineSet{}).
 		Watches(
@@ -73,22 +66,19 @@ func (r *MachineDeploymentReconciler) SetupWithManager(mgr ctrl.Manager, options
 			&handler.EnqueueRequestsFromMapFunc{ToRequests: handler.ToRequestsFunc(r.MachineSetToDeployments)},
 		).
 		WithOptions(options).
-		WithEventFilter(predicates.ResourceNotPaused(r.Log)).
 		Build(r)
+
 	if err != nil {
 		return errors.Wrap(err, "failed setting up with a controller manager")
 	}
 
-	err = c.Watch(
-		&source.Kind{Type: &clusterv1.Cluster{}},
-		&handler.EnqueueRequestsFromMapFunc{
-			ToRequests: clusterToMachineDeployments,
-		},
-		// TODO: should this wait for Cluster.Status.InfrastructureReady similar to Infra Machine resources?
-		predicates.ClusterUnpaused(r.Log),
-	)
+	// Add a watch on clusterv1.Cluster object for paused notifications.
+	clusterToMachineDeployments, err := util.ClusterToObjectsMapper(mgr.GetClient(), &clusterv1.MachineDeploymentList{}, mgr.GetScheme())
 	if err != nil {
-		return errors.Wrap(err, "failed to add Watch for Clusters to controller manager")
+		return err
+	}
+	if err := util.WatchOnClusterPaused(controller, clusterToMachineDeployments); err != nil {
+		return err
 	}
 
 	r.recorder = mgr.GetEventRecorderFor("machinedeployment-controller")
@@ -117,7 +107,7 @@ func (r *MachineDeploymentReconciler) Reconcile(req ctrl.Request) (_ ctrl.Result
 	}
 
 	// Return early if the object or Cluster is paused.
-	if annotations.IsPaused(cluster, deployment) {
+	if util.IsPaused(cluster, deployment) {
 		logger.Info("Reconciliation is paused for this object")
 		return ctrl.Result{}, nil
 	}
diff --git a/controllers/machinehealthcheck_controller.go b/controllers/machinehealthcheck_controller.go
index f6de9b728..5bd74f2a8 100644
--- a/controllers/machinehealthcheck_controller.go
+++ b/controllers/machinehealthcheck_controller.go
@@ -35,9 +35,7 @@ import (
 	clusterv1 "sigs.k8s.io/cluster-api/api/v1alpha3"
 	"sigs.k8s.io/cluster-api/controllers/remote"
 	"sigs.k8s.io/cluster-api/util"
-	"sigs.k8s.io/cluster-api/util/annotations"
 	"sigs.k8s.io/cluster-api/util/patch"
-	"sigs.k8s.io/cluster-api/util/predicates"
 	ctrl "sigs.k8s.io/controller-runtime"
 	"sigs.k8s.io/controller-runtime/pkg/cache"
 	"sigs.k8s.io/controller-runtime/pkg/client"
@@ -78,25 +76,20 @@ type MachineHealthCheckReconciler struct {
 func (r *MachineHealthCheckReconciler) SetupWithManager(mgr ctrl.Manager, options controller.Options) error {
 	controller, err := ctrl.NewControllerManagedBy(mgr).
 		For(&clusterv1.MachineHealthCheck{}).
+		Watches(
+			&source.Kind{Type: &clusterv1.Cluster{}},
+			&handler.EnqueueRequestsFromMapFunc{ToRequests: handler.ToRequestsFunc(r.clusterToMachineHealthCheck)},
+		).
 		Watches(
 			&source.Kind{Type: &clusterv1.Machine{}},
 			&handler.EnqueueRequestsFromMapFunc{ToRequests: handler.ToRequestsFunc(r.machineToMachineHealthCheck)},
 		).
 		WithOptions(options).
-		WithEventFilter(predicates.ResourceNotPaused(r.Log)).
 		Build(r)
+
 	if err != nil {
 		return errors.Wrap(err, "failed setting up with a controller manager")
 	}
-	err = controller.Watch(
-		&source.Kind{Type: &clusterv1.Cluster{}},
-		&handler.EnqueueRequestsFromMapFunc{ToRequests: handler.ToRequestsFunc(r.clusterToMachineHealthCheck)},
-		// TODO: should this wait for Cluster.Status.InfrastructureReady similar to Infra Machine resources?
-		predicates.ClusterUnpaused(r.Log),
-	)
-	if err != nil {
-		return errors.Wrap(err, "failed to add Watch for Clusters to controller manager")
-	}
 
 	// Add index to MachineHealthCheck for listing by Cluster Name
 	if err := mgr.GetCache().IndexField(&clusterv1.MachineHealthCheck{},
@@ -147,7 +140,7 @@ func (r *MachineHealthCheckReconciler) Reconcile(req ctrl.Request) (_ ctrl.Resul
 	}
 
 	// Return early if the object or Cluster is paused.
-	if annotations.IsPaused(cluster, m) {
+	if util.IsPaused(cluster, m) {
 		logger.Info("Reconciliation is paused for this object")
 		return ctrl.Result{}, nil
 	}
diff --git a/controllers/machineset_controller.go b/controllers/machineset_controller.go
index 2d3b16089..dedbe249a 100644
--- a/controllers/machineset_controller.go
+++ b/controllers/machineset_controller.go
@@ -38,10 +38,8 @@ import (
 	"sigs.k8s.io/cluster-api/controllers/noderefutil"
 	"sigs.k8s.io/cluster-api/controllers/remote"
 	"sigs.k8s.io/cluster-api/util"
-	"sigs.k8s.io/cluster-api/util/annotations"
 	utilconversion "sigs.k8s.io/cluster-api/util/conversion"
 	"sigs.k8s.io/cluster-api/util/patch"
-	"sigs.k8s.io/cluster-api/util/predicates"
 	ctrl "sigs.k8s.io/controller-runtime"
 	"sigs.k8s.io/controller-runtime/pkg/client"
 	"sigs.k8s.io/controller-runtime/pkg/controller"
@@ -77,12 +75,7 @@ type MachineSetReconciler struct {
 }
 
 func (r *MachineSetReconciler) SetupWithManager(mgr ctrl.Manager, options controller.Options) error {
-	clusterToMachineSets, err := util.ClusterToObjectsMapper(mgr.GetClient(), &clusterv1.MachineSetList{}, mgr.GetScheme())
-	if err != nil {
-		return err
-	}
-
-	c, err := ctrl.NewControllerManagedBy(mgr).
+	controller, err := ctrl.NewControllerManagedBy(mgr).
 		For(&clusterv1.MachineSet{}).
 		Owns(&clusterv1.Machine{}).
 		Watches(
@@ -90,20 +83,19 @@ func (r *MachineSetReconciler) SetupWithManager(mgr ctrl.Manager, options contro
 			&handler.EnqueueRequestsFromMapFunc{ToRequests: handler.ToRequestsFunc(r.MachineToMachineSets)},
 		).
 		WithOptions(options).
-		WithEventFilter(predicates.ResourceNotPaused(r.Log)).
 		Build(r)
+
 	if err != nil {
 		return errors.Wrap(err, "failed setting up with a controller manager")
 	}
 
-	err = c.Watch(
-		&source.Kind{Type: &clusterv1.Cluster{}},
-		&handler.EnqueueRequestsFromMapFunc{ToRequests: clusterToMachineSets},
-		// TODO: should this wait for Cluster.Status.InfrastructureReady similar to Infra Machine resources?
-		predicates.ClusterUnpaused(r.Log),
-	)
+	// Add a watch on clusterv1.Cluster object for paused notifications.
+	clusterToMachineSets, err := util.ClusterToObjectsMapper(mgr.GetClient(), &clusterv1.MachineSetList{}, mgr.GetScheme())
 	if err != nil {
-		return errors.Wrap(err, "failed to add Watch for Clusters to controller manager")
+		return err
+	}
+	if err := util.WatchOnClusterPaused(controller, clusterToMachineSets); err != nil {
+		return err
 	}
 
 	r.recorder = mgr.GetEventRecorderFor("machineset-controller")
@@ -132,7 +124,7 @@ func (r *MachineSetReconciler) Reconcile(req ctrl.Request) (ctrl.Result, error)
 	}
 
 	// Return early if the object or Cluster is paused.
-	if annotations.IsPaused(cluster, machineSet) {
+	if util.IsPaused(cluster, machineSet) {
 		logger.Info("Reconciliation is paused for this object")
 		return ctrl.Result{}, nil
 	}
diff --git a/controllers/remote/cluster.go b/controllers/remote/cluster.go
index 131bcf6a6..190098070 100644
--- a/controllers/remote/cluster.go
+++ b/controllers/remote/cluster.go
@@ -40,11 +40,12 @@ func NewClusterClient(ctx context.Context, c client.Client, cluster client.Objec
 	if err != nil {
 		return nil, errors.Wrapf(err, "failed to create client for Cluster %s/%s", cluster.Namespace, cluster.Name)
 	}
+
 	return ret, nil
 }
 
 // RESTConfig returns a configuration instance to be used with a Kubernetes client.
-func RESTConfig(ctx context.Context, c client.Reader, cluster client.ObjectKey) (*restclient.Config, error) {
+func RESTConfig(ctx context.Context, c client.Client, cluster client.ObjectKey) (*restclient.Config, error) {
 	kubeConfig, err := kcfg.FromSecret(ctx, c, cluster)
 	if err != nil {
 		return nil, errors.Wrapf(err, "failed to retrieve kubeconfig secret for Cluster %s/%s", cluster.Namespace, cluster.Name)
diff --git a/controlplane/kubeadm/controllers/controller.go b/controlplane/kubeadm/controllers/controller.go
index a601526cb..7831c6dec 100644
--- a/controlplane/kubeadm/controllers/controller.go
+++ b/controlplane/kubeadm/controllers/controller.go
@@ -36,9 +36,7 @@ import (
 	"sigs.k8s.io/cluster-api/controlplane/kubeadm/internal/machinefilters"
 	capierrors "sigs.k8s.io/cluster-api/errors"
 	"sigs.k8s.io/cluster-api/util"
-	"sigs.k8s.io/cluster-api/util/annotations"
 	"sigs.k8s.io/cluster-api/util/patch"
-	"sigs.k8s.io/cluster-api/util/predicates"
 	"sigs.k8s.io/cluster-api/util/secret"
 	ctrl "sigs.k8s.io/controller-runtime"
 	"sigs.k8s.io/controller-runtime/pkg/client"
@@ -65,30 +63,22 @@ type KubeadmControlPlaneReconciler struct {
 	controller controller.Controller
 	recorder   record.EventRecorder
 
-	managementCluster         internal.ManagementCluster
-	managementClusterUncached internal.ManagementCluster
+	managementCluster internal.ManagementCluster
 }
 
 func (r *KubeadmControlPlaneReconciler) SetupWithManager(mgr ctrl.Manager, options controller.Options) error {
 	c, err := ctrl.NewControllerManagedBy(mgr).
 		For(&controlplanev1.KubeadmControlPlane{}).
 		Owns(&clusterv1.Machine{}).
+		Watches(
+			&source.Kind{Type: &clusterv1.Cluster{}},
+			&handler.EnqueueRequestsFromMapFunc{ToRequests: handler.ToRequestsFunc(r.ClusterToKubeadmControlPlane)},
+		).
 		WithOptions(options).
-		WithEventFilter(predicates.ResourceNotPaused(r.Log)).
 		Build(r)
-	if err != nil {
-		return errors.Wrap(err, "failed setting up with a controller manager")
-	}
 
-	err = c.Watch(
-		&source.Kind{Type: &clusterv1.Cluster{}},
-		&handler.EnqueueRequestsFromMapFunc{
-			ToRequests: handler.ToRequestsFunc(r.ClusterToKubeadmControlPlane),
-		},
-		predicates.ClusterUnpausedAndInfrastructureReady(r.Log),
-	)
 	if err != nil {
-		return errors.Wrap(err, "failed adding Watch for Clusters to controller manager")
+		return errors.Wrap(err, "failed setting up with a controller manager")
 	}
 
 	r.scheme = mgr.GetScheme()
@@ -98,9 +88,6 @@ func (r *KubeadmControlPlaneReconciler) SetupWithManager(mgr ctrl.Manager, optio
 	if r.managementCluster == nil {
 		r.managementCluster = &internal.Management{Client: r.Client}
 	}
-	if r.managementClusterUncached == nil {
-		r.managementClusterUncached = &internal.Management{Client: mgr.GetAPIReader()}
-	}
 
 	return nil
 }
@@ -130,7 +117,7 @@ func (r *KubeadmControlPlaneReconciler) Reconcile(req ctrl.Request) (res ctrl.Re
 	}
 	logger = logger.WithValues("cluster", cluster.Name)
 
-	if annotations.IsPaused(cluster, kcp) {
+	if util.IsPaused(cluster, kcp) {
 		logger.Info("Reconciliation is paused for this object")
 		return ctrl.Result{}, nil
 	}
diff --git a/controlplane/kubeadm/controllers/scale.go b/controlplane/kubeadm/controllers/scale.go
index 8b2a4c95f..8cb1332f3 100644
--- a/controlplane/kubeadm/controllers/scale.go
+++ b/controlplane/kubeadm/controllers/scale.go
@@ -25,7 +25,6 @@ import (
 	clusterv1 "sigs.k8s.io/cluster-api/api/v1alpha3"
 	controlplanev1 "sigs.k8s.io/cluster-api/controlplane/kubeadm/api/v1alpha3"
 	"sigs.k8s.io/cluster-api/controlplane/kubeadm/internal"
-	"sigs.k8s.io/cluster-api/controlplane/kubeadm/internal/machinefilters"
 	capierrors "sigs.k8s.io/cluster-api/errors"
 	"sigs.k8s.io/cluster-api/util"
 	ctrl "sigs.k8s.io/controller-runtime"
@@ -34,20 +33,6 @@ import (
 func (r *KubeadmControlPlaneReconciler) initializeControlPlane(ctx context.Context, cluster *clusterv1.Cluster, kcp *controlplanev1.KubeadmControlPlane, controlPlane *internal.ControlPlane) (ctrl.Result, error) {
 	logger := controlPlane.Logger()
 
-	// Perform an uncached read of all the owned machines. This check is in place to make sure
-	// that the controller cache is not misbehaving and we end up initializing the cluster more than once.
-	ownedMachines, err := r.managementClusterUncached.GetMachinesForCluster(ctx, util.ObjectKey(cluster), machinefilters.OwnedControlPlaneMachines(kcp.Name))
-	if err != nil {
-		logger.Error(err, "failed to perform an uncached read of control plane machines for cluster")
-		return ctrl.Result{}, err
-	}
-	if len(ownedMachines) > 0 {
-		return ctrl.Result{}, errors.Errorf(
-			"control plane has already been initialized, found %d owned machine for cluster %s/%s: controller cache or management cluster is misbehaving",
-			len(ownedMachines), cluster.Namespace, cluster.Name,
-		)
-	}
-
 	bootstrapSpec := controlPlane.InitialControlPlaneConfig()
 	fd := controlPlane.FailureDomainWithFewestMachines()
 	if err := r.cloneConfigsAndGenerateMachine(ctx, cluster, kcp, bootstrapSpec, fd); err != nil {
diff --git a/controlplane/kubeadm/internal/cluster.go b/controlplane/kubeadm/internal/cluster.go
index 24cf31000..40a6be3d3 100644
--- a/controlplane/kubeadm/internal/cluster.go
+++ b/controlplane/kubeadm/internal/cluster.go
@@ -46,7 +46,7 @@ type ManagementCluster interface {
 
 // Management holds operations on the management cluster.
 type Management struct {
-	Client ctrlclient.Reader
+	Client ctrlclient.Client
 }
 
 // GetMachinesForCluster returns a list of machines that can be filtered or not.
diff --git a/docs/book/src/developer/providers/v1alpha2-to-v1alpha3.md b/docs/book/src/developer/providers/v1alpha2-to-v1alpha3.md
index 6c3b287a5..d51bbc2ff 100644
--- a/docs/book/src/developer/providers/v1alpha2-to-v1alpha3.md
+++ b/docs/book/src/developer/providers/v1alpha2-to-v1alpha3.md
@@ -162,45 +162,18 @@ outside of the existing module.
   }
   ```
 - Unless your controller is already watching Clusters, add a Watch to get notifications when Cluster.Spec.Paused field changes.
-  In most cases, `predicates.ClusterUnpaused` and `util.ClusterToObjectsMapper` can be used like in the example below:
+  In most cases, `util.WatchOnClusterPaused` and `util.ClusterToObjectsMapper` can be used like in the example below:
   ```go
   // Add a watch on clusterv1.Cluster object for paused notifications.
   clusterToObjectFunc, err := util.ClusterToObjectsMapper(mgr.GetClient(), <List object here>, mgr.GetScheme())
   if err != nil {
     return err
   }
-  err = controller.Watch(
-      &source.Kind{Type: &cluserv1.Cluster{}},
-      &handler.EnqueueRequestsFromMapFunc{
-          ToRequests: clusterToObjectFunc,
-      },
-      predicates.ClusterUnpaused(r.Log),
-  )
-  if err != nil {
+  if err := util.WatchOnClusterPaused(controller, clusterToObjectFunc); err != nil {
     return err
   }
   ```
   NB: You need to have `cluster.x-k8s.io/cluster-name` applied to all your objects for the mapper to function.
-- In some cases, you'll want to not just watch on Cluster.Spec.Paused changes, but also on
-  Cluster.Status.InfrastructureReady. For those cases `predicates.ClusterUnpausedAndInfrastructureReady` should be used
-  instead. 
-  ```go
-  // Add a watch on clusterv1.Cluster object for paused and infrastructureReady notifications.
-  clusterToObjectFunc, err := util.ClusterToObjectsMapper(mgr.GetClient(), <List object here>, mgr.GetScheme())
-  if err != nil {
-    return err
-  }
-  err = controller.Watch(
-        &source.Kind{Type: &cluserv1.Cluster{}},
-        &handler.EnqueueRequestsFromMapFunc{
-            ToRequests: clusterToObjectFunc,
-        },
-        predicates.ClusterUnpausedAndInfrastructureReady(r.Log),
-    )
-    if err != nil {
-      return err
-    }
-    ```
 
 ## [OPTIONAL] Support failure domains.
 
diff --git a/exp/controllers/machinepool_controller.go b/exp/controllers/machinepool_controller.go
index 9f8030317..1d1d64ffc 100644
--- a/exp/controllers/machinepool_controller.go
+++ b/exp/controllers/machinepool_controller.go
@@ -36,15 +36,11 @@ import (
 	capierrors "sigs.k8s.io/cluster-api/errors"
 	expv1 "sigs.k8s.io/cluster-api/exp/api/v1alpha3"
 	"sigs.k8s.io/cluster-api/util"
-	"sigs.k8s.io/cluster-api/util/annotations"
 	"sigs.k8s.io/cluster-api/util/patch"
-	"sigs.k8s.io/cluster-api/util/predicates"
 	ctrl "sigs.k8s.io/controller-runtime"
 	"sigs.k8s.io/controller-runtime/pkg/client"
 	"sigs.k8s.io/controller-runtime/pkg/controller"
 	"sigs.k8s.io/controller-runtime/pkg/controller/controllerutil"
-	"sigs.k8s.io/controller-runtime/pkg/handler"
-	"sigs.k8s.io/controller-runtime/pkg/source"
 )
 
 // +kubebuilder:rbac:groups=core,resources=events,verbs=get;list;watch;create;patch
@@ -66,30 +62,13 @@ type MachinePoolReconciler struct {
 }
 
 func (r *MachinePoolReconciler) SetupWithManager(mgr ctrl.Manager, options controller.Options) error {
-	clusterToMachinePools, err := util.ClusterToObjectsMapper(mgr.GetClient(), &expv1.MachinePoolList{}, mgr.GetScheme())
-	if err != nil {
-		return err
-	}
-
 	c, err := ctrl.NewControllerManagedBy(mgr).
 		For(&expv1.MachinePool{}).
 		WithOptions(options).
-		WithEventFilter(predicates.ResourceNotPaused(r.Log)).
 		Build(r)
 	if err != nil {
 		return errors.Wrap(err, "failed setting up with a controller manager")
 	}
-	err = c.Watch(
-		&source.Kind{Type: &clusterv1.Cluster{}},
-		&handler.EnqueueRequestsFromMapFunc{
-			ToRequests: clusterToMachinePools,
-		},
-		// TODO: should this wait for Cluster.Status.InfrastructureReady similar to Infra Machine resources?
-		predicates.ClusterUnpaused(r.Log),
-	)
-	if err != nil {
-		return errors.Wrap(err, "failed adding Watch for Cluster to controller manager")
-	}
 
 	r.controller = c
 	r.recorder = mgr.GetEventRecorderFor("machinepool-controller")
@@ -121,7 +100,7 @@ func (r *MachinePoolReconciler) Reconcile(req ctrl.Request) (_ ctrl.Result, rete
 	}
 
 	// Return early if the object or Cluster is paused.
-	if annotations.IsPaused(cluster, mp) {
+	if util.IsPaused(cluster, mp) {
 		logger.Info("Reconciliation is paused for this object")
 		return ctrl.Result{}, nil
 	}
diff --git a/exp/controllers/machinepool_controller_phases.go b/exp/controllers/machinepool_controller_phases.go
index ec0af7ddf..7ed61f1f0 100644
--- a/exp/controllers/machinepool_controller_phases.go
+++ b/exp/controllers/machinepool_controller_phases.go
@@ -33,7 +33,6 @@ import (
 	capierrors "sigs.k8s.io/cluster-api/errors"
 	expv1 "sigs.k8s.io/cluster-api/exp/api/v1alpha3"
 	"sigs.k8s.io/cluster-api/util"
-	"sigs.k8s.io/cluster-api/util/annotations"
 	"sigs.k8s.io/cluster-api/util/patch"
 	"sigs.k8s.io/controller-runtime/pkg/controller/controllerutil"
 	"sigs.k8s.io/controller-runtime/pkg/handler"
@@ -91,7 +90,7 @@ func (r *MachinePoolReconciler) reconcileExternal(ctx context.Context, cluster *
 	}
 
 	// if external ref is paused, return error.
-	if annotations.IsPaused(cluster, obj) {
+	if util.IsPaused(cluster, obj) {
 		logger.V(3).Info("External object referenced is paused")
 		return external.ReconcileOutput{Paused: true}, nil
 	}
diff --git a/go.mod b/go.mod
index 48337f501..e58637016 100644
--- a/go.mod
+++ b/go.mod
@@ -11,9 +11,12 @@ require (
 	github.com/evanphx/json-patch v4.5.0+incompatible
 	github.com/go-logr/logr v0.1.0
 	github.com/gogo/protobuf v1.3.1
+	github.com/golang/groupcache v0.0.0-20200121045136-8c9f03a8e57e // indirect
 	github.com/google/go-github v17.0.0+incompatible
 	github.com/google/go-querystring v1.0.0 // indirect
 	github.com/google/gofuzz v1.1.0
+	github.com/hashicorp/golang-lru v0.5.4 // indirect
+	github.com/imdario/mergo v0.3.8 // indirect
 	github.com/onsi/ginkgo v1.12.0
 	github.com/onsi/gomega v1.9.0
 	github.com/opencontainers/go-digest v1.0.0-rc1 // indirect
@@ -24,8 +27,10 @@ require (
 	github.com/spf13/pflag v1.0.5
 	github.com/spf13/viper v1.6.2
 	go.etcd.io/etcd v0.0.0-20191023171146-3cf2f69b5738
+	golang.org/x/crypto v0.0.0-20200302210943-78000ba7a073 // indirect
 	golang.org/x/net v0.0.0-20200301022130-244492dfa37a
 	golang.org/x/oauth2 v0.0.0-20200107190931-bf48bf16ab8d
+	golang.org/x/time v0.0.0-20191024005414-555d28b269f0 // indirect
 	google.golang.org/appengine v1.6.1 // indirect
 	google.golang.org/grpc v1.26.0
 	k8s.io/api v0.17.2
diff --git a/go.sum b/go.sum
index c468b50c2..e129e3537 100644
--- a/go.sum
+++ b/go.sum
@@ -185,8 +185,9 @@ github.com/gogo/protobuf v1.3.1/go.mod h1:SlYgWuQ5SjCEi6WLHjHCa1yvBfUnHcTbrrZtXP
 github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b h1:VKtxabqXZkF25pY9ekfRL6a582T4P37/31XEstQ5p58=
 github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q=
 github.com/golang/groupcache v0.0.0-20160516000752-02826c3e7903/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
-github.com/golang/groupcache v0.0.0-20190129154638-5b532d6fd5ef h1:veQD95Isof8w9/WXiA+pa3tz3fJXkt5B7QaRBrM62gk=
 github.com/golang/groupcache v0.0.0-20190129154638-5b532d6fd5ef/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
+github.com/golang/groupcache v0.0.0-20200121045136-8c9f03a8e57e h1:1r7pUrabqp18hOBcwBwiTsbnFeTZHV9eER/QT5JVZxY=
+github.com/golang/groupcache v0.0.0-20200121045136-8c9f03a8e57e/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
 github.com/golang/mock v1.1.1/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A=
 github.com/golang/mock v1.2.0/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A=
 github.com/golang/protobuf v0.0.0-20161109072736-4bd1920723d7/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
@@ -239,15 +240,17 @@ github.com/grpc-ecosystem/grpc-gateway v1.9.5/go.mod h1:vNeuVxBJEsws4ogUvrchl83t
 github.com/hashicorp/go-syslog v1.0.0/go.mod h1:qPfqrKkXGihmCqbJM2mZgkZGvKG1dFdvsLplgctolz4=
 github.com/hashicorp/golang-lru v0.0.0-20180201235237-0fb14efe8c47/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8=
 github.com/hashicorp/golang-lru v0.5.0/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8=
-github.com/hashicorp/golang-lru v0.5.1 h1:0hERBMJE1eitiLkihrMvRVBYAkpHzc/J3QdDN+dAcgU=
 github.com/hashicorp/golang-lru v0.5.1/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8=
+github.com/hashicorp/golang-lru v0.5.4 h1:YDjusn29QI/Das2iO9M0BHnIbxPeyuCHsjMW+lJfyTc=
+github.com/hashicorp/golang-lru v0.5.4/go.mod h1:iADmTwqILo4mZ8BN3D2Q6+9jd8WM5uGBxy+E8yxSoD4=
 github.com/hashicorp/hcl v1.0.0 h1:0Anlzjpi4vEasTeNFn2mLJgTSwt0+6sfsiTG8qcWGx4=
 github.com/hashicorp/hcl v1.0.0/go.mod h1:E5yfLk+7swimpb2L/Alb/PJmXilQ/rhwaUYs4T20WEQ=
 github.com/hpcloud/tail v1.0.0 h1:nfCOvKYfkgYP8hkirhJocXT2+zOD8yUNjXaWfTlyFKI=
 github.com/hpcloud/tail v1.0.0/go.mod h1:ab1qPbhIpdTxEkNHXyeSf5vhxWSCs/tWer42PpOxQnU=
 github.com/imdario/mergo v0.3.5/go.mod h1:2EnlNZ0deacrJVfApfmtdGgDfMuh/nq6Ok1EcJh5FfA=
-github.com/imdario/mergo v0.3.6 h1:xTNEAn+kxVO7dTZGu0CegyqKZmoWFI0rF8UxjlB2d28=
 github.com/imdario/mergo v0.3.6/go.mod h1:2EnlNZ0deacrJVfApfmtdGgDfMuh/nq6Ok1EcJh5FfA=
+github.com/imdario/mergo v0.3.8 h1:CGgOkSJeqMRmt0D9XLWExdT4m4F1vd3FV3VPt+0VxkQ=
+github.com/imdario/mergo v0.3.8/go.mod h1:2EnlNZ0deacrJVfApfmtdGgDfMuh/nq6Ok1EcJh5FfA=
 github.com/inconshreveable/mousetrap v1.0.0 h1:Z8tu5sraLXCXIcARxBp/8cbvlwVa7Z1NHg9XEKhtSvM=
 github.com/inconshreveable/mousetrap v1.0.0/go.mod h1:PxqpIevigyE2G7u3NXJIT2ANytuPF1OarO4DADm73n8=
 github.com/jimstudt/http-authentication v0.0.0-20140401203705-3eca13d6893a/go.mod h1:wK6yTYYcgjHE1Z1QtXACPDjcFJyBskHEdagmnq3vsP8=
@@ -462,8 +465,9 @@ golang.org/x/crypto v0.0.0-20190605123033-f99c8df09eb5/go.mod h1:yigFU9vqHzYiE8U
 golang.org/x/crypto v0.0.0-20190611184440-5c40567a22f8/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
 golang.org/x/crypto v0.0.0-20190617133340-57b3e21c3d56/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
 golang.org/x/crypto v0.0.0-20190820162420-60c769a6c586/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
-golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550 h1:ObdrDkeb4kJdCP557AjRjq69pTHfNouLtWZG7j9rPN8=
 golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
+golang.org/x/crypto v0.0.0-20200302210943-78000ba7a073 h1:xMPOj6Pz6UipU1wXLkrtqpHbR0AVFnyPEQq/wRWz9lM=
+golang.org/x/crypto v0.0.0-20200302210943-78000ba7a073/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
 golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
 golang.org/x/exp v0.0.0-20190125153040-c74c464bbbf2/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
 golang.org/x/exp v0.0.0-20190312203227-4b39c73a6495/go.mod h1:ZjyILWgesfNpC6sMxTJOJm9Kp84zZh5NQWvqDGG3Qr8=
@@ -538,8 +542,9 @@ golang.org/x/text v0.3.2 h1:tW2bmiBqwgJj/UpqtC8EpXEZVYOwU0yG4iWbprSVAcs=
 golang.org/x/text v0.3.2/go.mod h1:bEr9sfX3Q8Zfm5fL9x+3itogRgK3+ptLWKqgva+5dAk=
 golang.org/x/time v0.0.0-20180412165947-fbb02b2291d2/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
 golang.org/x/time v0.0.0-20181108054448-85acf8d2951c/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
-golang.org/x/time v0.0.0-20190308202827-9d24e82272b4 h1:SvFZT6jyqRaOeXpc5h/JSfZenJ2O330aBsf7JfSUXmQ=
 golang.org/x/time v0.0.0-20190308202827-9d24e82272b4/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
+golang.org/x/time v0.0.0-20191024005414-555d28b269f0 h1:/5xXl8Y5W96D+TtHSlonuFqGHIWVuyCkGJLwGh9JJFs=
+golang.org/x/time v0.0.0-20191024005414-555d28b269f0/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
 golang.org/x/tools v0.0.0-20180221164845-07fd8470d635/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
 golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
 golang.org/x/tools v0.0.0-20181011042414-1f849cf54d09/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
diff --git a/util/annotations/paused.go b/util/annotations/paused.go
deleted file mode 100644
index acb817a05..000000000
--- a/util/annotations/paused.go
+++ /dev/null
@@ -1,40 +0,0 @@
-/*
-Copyright 2020 The Kubernetes Authors.
-
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
-    http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-*/
-
-package annotations
-
-import (
-	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
-	clusterv1 "sigs.k8s.io/cluster-api/api/v1alpha3"
-)
-
-// IsPaused returns true if the Cluster is paused or the object has the `paused` annotation.
-func IsPaused(cluster *clusterv1.Cluster, o metav1.Object) bool {
-	if cluster.Spec.Paused {
-		return true
-	}
-	return HasPausedAnnotation(o)
-}
-
-// HasPausedAnnotation returns true if the object has the `paused` annotation.
-func HasPausedAnnotation(o metav1.Object) bool {
-	annotations := o.GetAnnotations()
-	if annotations == nil {
-		return false
-	}
-	_, ok := annotations[clusterv1.PausedAnnotation]
-	return ok
-}
diff --git a/util/kubeconfig/kubeconfig.go b/util/kubeconfig/kubeconfig.go
index a5ffc11f2..32a51d7a2 100644
--- a/util/kubeconfig/kubeconfig.go
+++ b/util/kubeconfig/kubeconfig.go
@@ -40,7 +40,7 @@ var (
 )
 
 // FromSecret fetches the Kubeconfig for a Cluster.
-func FromSecret(ctx context.Context, c client.Reader, cluster client.ObjectKey) ([]byte, error) {
+func FromSecret(ctx context.Context, c client.Client, cluster client.ObjectKey) ([]byte, error) {
 	out, err := secret.Get(ctx, c, cluster, secret.Kubeconfig)
 	if err != nil {
 		return nil, err
diff --git a/util/predicates/cluster_predicates.go b/util/predicates/cluster_predicates.go
deleted file mode 100644
index 06b76ee96..000000000
--- a/util/predicates/cluster_predicates.go
+++ /dev/null
@@ -1,195 +0,0 @@
-/*
-Copyright 2020 The Kubernetes Authors.
-
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
-    http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-*/
-
-package predicates
-
-import (
-	"github.com/go-logr/logr"
-	clusterv1 "sigs.k8s.io/cluster-api/api/v1alpha3"
-	"sigs.k8s.io/controller-runtime/pkg/event"
-	"sigs.k8s.io/controller-runtime/pkg/predicate"
-)
-
-// ClusterCreateInfraReady returns a predicate that returns true for a create event when a cluster has Status.InfrastructureReady set as true
-// it also returns true if the resource provided is not a Cluster to allow for use with controller-runtime NewControllerManagedBy
-func ClusterCreateInfraReady(logger logr.Logger) predicate.Funcs {
-	log := logger.WithValues("predicate", "ClusterCreateInfraReady")
-	return predicate.Funcs{
-		CreateFunc: func(e event.CreateEvent) bool {
-			log = log.WithValues("eventType", "create")
-
-			c, ok := e.Object.(*clusterv1.Cluster)
-			if !ok {
-
-				log.V(4).Info("Expected Cluster", "type", e.Object.GetObjectKind().GroupVersionKind().String())
-				return false
-			}
-			log = log.WithValues("namespace", c.Namespace, "cluster", c.Name)
-
-			// Only need to trigger a reconcile if the Cluster.Status.InfrastructureReady is true
-			if c.Status.InfrastructureReady {
-				log.V(4).Info("Cluster infrastructure is ready, allowing further processing")
-				return true
-			}
-
-			log.V(4).Info("Cluster infrastructure is not ready, blocking further processing")
-			return false
-		},
-		UpdateFunc:  func(e event.UpdateEvent) bool { return false },
-		DeleteFunc:  func(e event.DeleteEvent) bool { return false },
-		GenericFunc: func(e event.GenericEvent) bool { return false },
-	}
-}
-
-// ClusterCreateNotPaused returns a predicate that returns true for a create event when a cluster has Spec.Paused set as false
-// it also returns true if the resource provided is not a Cluster to allow for use with controller-runtime NewControllerManagedBy
-func ClusterCreateNotPaused(logger logr.Logger) predicate.Funcs {
-	log := logger.WithValues("predicate", "ClusterCreateNotPaused")
-	return predicate.Funcs{
-		CreateFunc: func(e event.CreateEvent) bool {
-			log = log.WithValues("eventType", "create")
-
-			c, ok := e.Object.(*clusterv1.Cluster)
-			if !ok {
-
-				log.V(4).Info("Expected Cluster", "type", e.Object.GetObjectKind().GroupVersionKind().String())
-				return false
-			}
-			log = log.WithValues("namespace", c.Namespace, "cluster", c.Name)
-
-			// Only need to trigger a reconcile if the Cluster.Spec.Paused is false
-			if !c.Spec.Paused {
-				log.V(4).Info("Cluster is not paused, allowing further processing")
-				return true
-			}
-
-			log.V(4).Info("Cluster is not paused, blocking further processing")
-			return false
-		},
-		UpdateFunc:  func(e event.UpdateEvent) bool { return false },
-		DeleteFunc:  func(e event.DeleteEvent) bool { return false },
-		GenericFunc: func(e event.GenericEvent) bool { return false },
-	}
-}
-
-// ClusterUpdateInfraReady returns a predicate that returns true for an update event when a cluster has Status.InfrastructureReady changed from false to true
-// it also returns true if the resource provided is not a Cluster to allow for use with controller-runtime NewControllerManagedBy
-func ClusterUpdateInfraReady(logger logr.Logger) predicate.Funcs {
-	log := logger.WithValues("predicate", "ClusterUpdateInfraReady")
-	return predicate.Funcs{
-		UpdateFunc: func(e event.UpdateEvent) bool {
-			log = log.WithValues("eventType", "update")
-
-			oldCluster, ok := e.ObjectOld.(*clusterv1.Cluster)
-			if !ok {
-
-				log.V(4).Info("Expected Cluster", "type", e.ObjectOld.GetObjectKind().GroupVersionKind().String())
-				return false
-			}
-			log = log.WithValues("namespace", oldCluster.Namespace, "cluster", oldCluster.Name)
-
-			newCluster := e.ObjectNew.(*clusterv1.Cluster)
-
-			if !oldCluster.Status.InfrastructureReady && newCluster.Status.InfrastructureReady {
-				log.V(4).Info("Cluster infrastructure became ready, allowing further processing")
-				return true
-			}
-
-			log.V(4).Info("Cluster infrastructure did not become ready, blocking further processing")
-			return false
-		},
-		CreateFunc:  func(e event.CreateEvent) bool { return false },
-		DeleteFunc:  func(e event.DeleteEvent) bool { return false },
-		GenericFunc: func(e event.GenericEvent) bool { return false },
-	}
-}
-
-// ClusterUpdateUnpaused returns a predicate that returns true for an update event when a cluster has Spec.Paused changed from true to false
-// it also returns true if the resource provided is not a Cluster to allow for use with controller-runtime NewControllerManagedBy
-func ClusterUpdateUnpaused(logger logr.Logger) predicate.Funcs {
-	log := logger.WithValues("predicate", "ClusterUpdateUnpaused")
-	return predicate.Funcs{
-		UpdateFunc: func(e event.UpdateEvent) bool {
-			log = log.WithValues("eventType", "update")
-
-			oldCluster, ok := e.ObjectOld.(*clusterv1.Cluster)
-			if !ok {
-
-				log.V(4).Info("Expected Cluster", "type", e.ObjectOld.GetObjectKind().GroupVersionKind().String())
-				return false
-			}
-			log = log.WithValues("namespace", oldCluster.Namespace, "cluster", oldCluster.Name)
-
-			newCluster := e.ObjectNew.(*clusterv1.Cluster)
-
-			if oldCluster.Spec.Paused && !newCluster.Spec.Paused {
-				log.V(4).Info("Cluster was unpaused, allowing further processing")
-				return true
-			}
-
-			log.V(4).Info("Cluster was not unpaused, blocking further processing")
-			return false
-		},
-		CreateFunc:  func(e event.CreateEvent) bool { return false },
-		DeleteFunc:  func(e event.DeleteEvent) bool { return false },
-		GenericFunc: func(e event.GenericEvent) bool { return false },
-	}
-}
-
-// ClusterUnpaused returns a Predicate that returns true on Cluster creation events where Cluster.Spec.Paused is false
-// and Update events when Cluster.Spec.Paused transitions to false.
-// This implements a common requirement for many cluster-api and provider controllers (such as Cluster Infrastructure
-// controllers) to resume reconciliation when the Cluster is unpaused.
-// Example use:
-//  err := controller.Watch(
-//      &source.Kind{Type: &clusterv1.Cluster{}},
-//      &handler.EnqueueRequestsFromMapFunc{
-//          ToRequests: clusterToMachines,
-//      },
-//      predicates.ClusterUnpaused(r.Log),
-//  )
-func ClusterUnpaused(logger logr.Logger) predicate.Funcs {
-	log := logger.WithValues("predicate", "ClusterUnpaused")
-
-	// Use any to ensure we process either create or update events we care about
-	return Any(log, ClusterCreateNotPaused(log), ClusterUpdateUnpaused(log))
-}
-
-// ClusterUnpausedAndInfrastructureReady returns a Predicate that returns true on Cluster creation events where
-// both Cluster.Spec.Paused is false and Cluster.Status.InfrastructureReady is true and Update events when
-// either Cluster.Spec.Paused transitions to false or Cluster.Status.InfrastructureReady transitions to true.
-// This implements a common requirement for some cluster-api and provider controllers (such as Machine Infrastructure
-// controllers) to resume reconciliation when the Cluster is unpaused and when the infrastructure becomes ready.
-// Example use:
-//  err := controller.Watch(
-//      &source.Kind{Type: &clusterv1.Cluster{}},
-//      &handler.EnqueueRequestsFromMapFunc{
-//          ToRequests: clusterToMachines,
-//      },
-//      predicates.ClusterUnpausedAndInfrastructureReady(r.Log),
-//  )
-func ClusterUnpausedAndInfrastructureReady(logger logr.Logger) predicate.Funcs {
-	log := logger.WithValues("predicate", "ClusterUnpausedAndInfrastructureReady")
-
-	// Only continue processing create events if both not paused and infrastructure is ready
-	createPredicates := All(log, ClusterCreateNotPaused(log), ClusterCreateInfraReady(log))
-
-	// Process update events if either Cluster is unpaused or infrastructure becomes ready
-	updatePredicates := Any(log, ClusterUpdateUnpaused(log), ClusterUpdateInfraReady(log))
-
-	// Use any to ensure we process either create or update events we care about
-	return Any(log, createPredicates, updatePredicates)
-}
diff --git a/util/predicates/generic_predicates.go b/util/predicates/generic_predicates.go
deleted file mode 100644
index 6fac48261..000000000
--- a/util/predicates/generic_predicates.go
+++ /dev/null
@@ -1,164 +0,0 @@
-/*
-Copyright 2020 The Kubernetes Authors.
-
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
-    http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-*/
-
-package predicates
-
-import (
-	"strings"
-
-	"github.com/go-logr/logr"
-	v1 "k8s.io/apimachinery/pkg/apis/meta/v1"
-	"k8s.io/apimachinery/pkg/runtime"
-	"sigs.k8s.io/cluster-api/util/annotations"
-
-	"sigs.k8s.io/controller-runtime/pkg/event"
-	"sigs.k8s.io/controller-runtime/pkg/predicate"
-)
-
-// All returns a predicate that returns true only if all given predicates return true
-func All(logger logr.Logger, predicates ...predicate.Funcs) predicate.Funcs {
-	log := logger.WithValues("predicateAggregation", "All")
-	return predicate.Funcs{
-		UpdateFunc: func(e event.UpdateEvent) bool {
-			for _, p := range predicates {
-				if !p.UpdateFunc(e) {
-					log.V(4).Info("One of the provided predicates returned false, blocking further processing")
-					return false
-				}
-			}
-			log.V(4).Info("All provided predicates returned true, allowing further processing")
-			return true
-		},
-		CreateFunc: func(e event.CreateEvent) bool {
-			for _, p := range predicates {
-				if !p.CreateFunc(e) {
-					log.V(4).Info("One of the provided predicates returned false, blocking further processing")
-					return false
-				}
-			}
-			log.V(4).Info("All provided predicates returned true, allowing further processing")
-			return true
-		},
-		DeleteFunc: func(e event.DeleteEvent) bool {
-			for _, p := range predicates {
-				if !p.DeleteFunc(e) {
-					log.V(4).Info("One of the provided predicates returned false, blocking further processing")
-					return false
-				}
-			}
-			log.V(4).Info("All provided predicates returned true, allowing further processing")
-			return true
-		},
-		GenericFunc: func(e event.GenericEvent) bool {
-			for _, p := range predicates {
-				if !p.GenericFunc(e) {
-					log.V(4).Info("One of the provided predicates returned false, blocking further processing")
-					return false
-				}
-			}
-			log.V(4).Info("All provided predicates returned true, allowing further processing")
-			return true
-		},
-	}
-}
-
-// Any returns a predicate that returns true only if any given predicate returns true
-func Any(logger logr.Logger, predicates ...predicate.Funcs) predicate.Funcs {
-	log := logger.WithValues("predicateAggregation", "Any")
-	return predicate.Funcs{
-		UpdateFunc: func(e event.UpdateEvent) bool {
-			for _, p := range predicates {
-				if p.UpdateFunc(e) {
-					log.V(4).Info("One of the provided predicates returned true, allowing further processing")
-					return true
-				}
-			}
-			log.V(4).Info("All of the provided predicates returned false, blocking further processing")
-			return false
-		},
-		CreateFunc: func(e event.CreateEvent) bool {
-			for _, p := range predicates {
-				if p.CreateFunc(e) {
-					log.V(4).Info("One of the provided predicates returned true, allowing further processing")
-					return true
-				}
-			}
-			log.V(4).Info("All of the provided predicates returned false, blocking further processing")
-			return false
-		},
-		DeleteFunc: func(e event.DeleteEvent) bool {
-			for _, p := range predicates {
-				if p.DeleteFunc(e) {
-					log.V(4).Info("One of the provided predicates returned true, allowing further processing")
-					return true
-				}
-			}
-			log.V(4).Info("All of the provided predicates returned false, blocking further processing")
-			return false
-		},
-		GenericFunc: func(e event.GenericEvent) bool {
-			for _, p := range predicates {
-				if p.GenericFunc(e) {
-					log.V(4).Info("One of the provided predicates returned true, allowing further processing")
-					return true
-				}
-			}
-			log.V(4).Info("All of the provided predicates returned false, blocking further processing")
-			return false
-		},
-	}
-}
-
-// ResourceNotPaused returns a Predicate that returns true only if the provided resource does not contain the
-// paused annotation.
-// This implements a common requirement for all cluster-api and provider controllers skip reconciliation when the paused
-// annotation is present for a resource.
-// Example use:
-//	func (r *MyReconciler) SetupWithManager(mgr ctrl.Manager, options controller.Options) error {
-//		controller, err := ctrl.NewControllerManagedBy(mgr).
-//			For(&v1.MyType{}).
-//			WithOptions(options).
-//			WithEventFilter(util.ResourceNotPaused(r.Log)).
-//			Build(r)
-//		return err
-//	}
-func ResourceNotPaused(logger logr.Logger) predicate.Funcs {
-	return predicate.Funcs{
-		UpdateFunc: func(e event.UpdateEvent) bool {
-			return processIfNotPaused(logger.WithValues("predicate", "updateEvent"), e.ObjectNew, e.MetaNew)
-		},
-		CreateFunc: func(e event.CreateEvent) bool {
-			return processIfNotPaused(logger.WithValues("predicate", "createEvent"), e.Object, e.Meta)
-		},
-		DeleteFunc: func(e event.DeleteEvent) bool {
-			return processIfNotPaused(logger.WithValues("predicate", "deleteEvent"), e.Object, e.Meta)
-		},
-		GenericFunc: func(e event.GenericEvent) bool {
-			return processIfNotPaused(logger.WithValues("predicate", "genericEvent"), e.Object, e.Meta)
-		},
-	}
-}
-
-func processIfNotPaused(logger logr.Logger, obj runtime.Object, meta v1.Object) bool {
-	kind := strings.ToLower(obj.GetObjectKind().GroupVersionKind().Kind)
-	log := logger.WithValues("namespace", meta.GetNamespace(), kind, meta.GetName())
-	if annotations.HasPausedAnnotation(meta) {
-		log.V(4).Info("Resource is paused, will not attempt to map resource")
-		return false
-	}
-	log.V(4).Info("Resource is not paused, will attempt to map resource")
-	return true
-}
diff --git a/util/secret/secret.go b/util/secret/secret.go
index 5af9fd9f5..a17b3f430 100644
--- a/util/secret/secret.go
+++ b/util/secret/secret.go
@@ -28,13 +28,13 @@ import (
 
 // Get retrieves the specified Secret (if any) from the given
 // cluster name and namespace.
-func Get(ctx context.Context, c client.Reader, cluster client.ObjectKey, purpose Purpose) (*corev1.Secret, error) {
+func Get(ctx context.Context, c client.Client, cluster client.ObjectKey, purpose Purpose) (*corev1.Secret, error) {
 	return GetFromNamespacedName(ctx, c, cluster, purpose)
 }
 
 // GetFromNamespacedName retrieves the specified Secret (if any) from the given
 // cluster name and namespace.
-func GetFromNamespacedName(ctx context.Context, c client.Reader, clusterName client.ObjectKey, purpose Purpose) (*corev1.Secret, error) {
+func GetFromNamespacedName(ctx context.Context, c client.Client, clusterName client.ObjectKey, purpose Purpose) (*corev1.Secret, error) {
 	secret := &corev1.Secret{}
 	secretKey := client.ObjectKey{
 		Namespace: clusterName.Namespace,
diff --git a/util/util.go b/util/util.go
index e455b8af3..efce5c790 100644
--- a/util/util.go
+++ b/util/util.go
@@ -37,16 +37,15 @@ import (
 	"k8s.io/apimachinery/pkg/runtime"
 	"k8s.io/apimachinery/pkg/runtime/schema"
 	"k8s.io/apimachinery/pkg/version"
-	"k8s.io/klog/klogr"
 	clusterv1 "sigs.k8s.io/cluster-api/api/v1alpha3"
-	"sigs.k8s.io/cluster-api/util/annotations"
 	"sigs.k8s.io/cluster-api/util/container"
-	"sigs.k8s.io/cluster-api/util/predicates"
 	ctrl "sigs.k8s.io/controller-runtime"
 	"sigs.k8s.io/controller-runtime/pkg/client"
 	"sigs.k8s.io/controller-runtime/pkg/client/apiutil"
 	"sigs.k8s.io/controller-runtime/pkg/controller"
+	"sigs.k8s.io/controller-runtime/pkg/event"
 	"sigs.k8s.io/controller-runtime/pkg/handler"
+	"sigs.k8s.io/controller-runtime/pkg/predicate"
 	"sigs.k8s.io/controller-runtime/pkg/reconcile"
 	"sigs.k8s.io/controller-runtime/pkg/source"
 )
@@ -434,15 +433,23 @@ func HasOwner(refList []metav1.OwnerReference, apiVersion string, kinds []string
 	return false
 }
 
-var (
-	// IsPaused returns true if the Cluster is paused or the object has the `paused` annotation.
-	// Deprecated: use util/annotations/IsPaused instead
-	IsPaused = annotations.IsPaused
+// IsPaused returns true if the Cluster is paused or the object has the `paused` annotation.
+func IsPaused(cluster *clusterv1.Cluster, o metav1.Object) bool {
+	if cluster.Spec.Paused {
+		return true
+	}
+	return HasPausedAnnotation(o)
+}
 
-	// HasPausedAnnotation returns true if the object has the `paused` annotation.
-	// Deprecated: use util/annotations/HasPausedAnnotation instead
-	HasPausedAnnotation = annotations.HasPausedAnnotation
-)
+// HasPausedAnnotation returns true if the object has the `paused` annotation.
+func HasPausedAnnotation(o metav1.Object) bool {
+	annotations := o.GetAnnotations()
+	if annotations == nil {
+		return false
+	}
+	_, ok := annotations[clusterv1.PausedAnnotation]
+	return ok
+}
 
 // GetCRDWithContract retrieves a list of CustomResourceDefinitions from using controller-runtime Client,
 // filtering with the `contract` label passed in.
@@ -498,16 +505,19 @@ func (o MachinesByCreationTimestamp) Less(i, j int) bool {
 // WatchOnClusterPaused adds a conditional watch to the controlled given as input
 // that sends watch notifications on any create or delete, and only updates
 // that toggle Cluster.Spec.Cluster.
-// Deprecated: Instead add the Watch directly and use predicates.ClusterUnpaused or
-// predicates.ClusterUnpausedAndInfrastructureReady depending on your use case.
 func WatchOnClusterPaused(c controller.Controller, mapFunc handler.Mapper) error {
-	log := klogr.New().WithName("WatchOnClusterPaused")
 	return c.Watch(
 		&source.Kind{Type: &clusterv1.Cluster{}},
 		&handler.EnqueueRequestsFromMapFunc{
 			ToRequests: mapFunc,
 		},
-		predicates.ClusterUnpaused(log),
+		predicate.Funcs{
+			UpdateFunc: func(e event.UpdateEvent) bool {
+				oldCluster := e.ObjectOld.(*clusterv1.Cluster)
+				newCluster := e.ObjectNew.(*clusterv1.Cluster)
+				return oldCluster.Spec.Paused && !newCluster.Spec.Paused
+			},
+		},
 	)
 }
 


Test patch:
diff --git a/controllers/machine_controller_test.go b/controllers/machine_controller_test.go
index ee4d1574b..81b2fa526 100644
--- a/controllers/machine_controller_test.go
+++ b/controllers/machine_controller_test.go
@@ -338,7 +338,7 @@ func TestReconcileRequest(t *testing.T) {
 				ObjectMeta: metav1.ObjectMeta{
 					Name:       "created",
 					Namespace:  "default",
-					Finalizers: []string{clusterv1.MachineFinalizer},
+					Finalizers: []string{clusterv1.MachineFinalizer, metav1.FinalizerDeleteDependents},
 				},
 				Spec: clusterv1.MachineSpec{
 					ClusterName: "test-cluster",
@@ -365,7 +365,7 @@ func TestReconcileRequest(t *testing.T) {
 				ObjectMeta: metav1.ObjectMeta{
 					Name:       "updated",
 					Namespace:  "default",
-					Finalizers: []string{clusterv1.MachineFinalizer},
+					Finalizers: []string{clusterv1.MachineFinalizer, metav1.FinalizerDeleteDependents},
 				},
 				Spec: clusterv1.MachineSpec{
 					ClusterName: "test-cluster",
@@ -395,7 +395,7 @@ func TestReconcileRequest(t *testing.T) {
 					Labels: map[string]string{
 						clusterv1.MachineControlPlaneLabelName: "",
 					},
-					Finalizers:        []string{clusterv1.MachineFinalizer},
+					Finalizers:        []string{clusterv1.MachineFinalizer, metav1.FinalizerDeleteDependents},
 					DeletionTimestamp: &time,
 				},
 				Spec: clusterv1.MachineSpec{
@@ -575,7 +575,7 @@ func TestRemoveMachineFinalizerAfterDeleteReconcile(t *testing.T) {
 		ObjectMeta: metav1.ObjectMeta{
 			Name:              "delete123",
 			Namespace:         "default",
-			Finalizers:        []string{clusterv1.MachineFinalizer},
+			Finalizers:        []string{clusterv1.MachineFinalizer, metav1.FinalizerDeleteDependents},
 			DeletionTimestamp: &dt,
 		},
 		Spec: clusterv1.MachineSpec{
@@ -597,9 +597,8 @@ func TestRemoveMachineFinalizerAfterDeleteReconcile(t *testing.T) {
 	_, err := mr.Reconcile(reconcile.Request{NamespacedName: key})
 	g.Expect(err).ToNot(HaveOccurred())
 
-	var actual clusterv1.Machine
-	g.Expect(mr.Client.Get(ctx, key, &actual)).To(Succeed())
-	g.Expect(actual.ObjectMeta.Finalizers).To(BeEmpty())
+	g.Expect(mr.Client.Get(ctx, key, m)).To(Succeed())
+	g.Expect(m.ObjectMeta.Finalizers).To(Equal([]string{metav1.FinalizerDeleteDependents}))
 }
 
 func TestReconcileMetrics(t *testing.T) {
@@ -804,7 +803,7 @@ func TestIsDeleteNodeAllowed(t *testing.T) {
 				ObjectMeta: metav1.ObjectMeta{
 					Name:       "created",
 					Namespace:  "default",
-					Finalizers: []string{clusterv1.MachineFinalizer},
+					Finalizers: []string{clusterv1.MachineFinalizer, metav1.FinalizerDeleteDependents},
 				},
 				Spec: clusterv1.MachineSpec{
 					ClusterName:       "test-cluster",
@@ -822,7 +821,7 @@ func TestIsDeleteNodeAllowed(t *testing.T) {
 				ObjectMeta: metav1.ObjectMeta{
 					Name:       "created",
 					Namespace:  "default",
-					Finalizers: []string{clusterv1.MachineFinalizer},
+					Finalizers: []string{clusterv1.MachineFinalizer, metav1.FinalizerDeleteDependents},
 				},
 				Spec: clusterv1.MachineSpec{
 					ClusterName:       "test-cluster",
@@ -848,7 +847,7 @@ func TestIsDeleteNodeAllowed(t *testing.T) {
 						clusterv1.ClusterLabelName:             "test",
 						clusterv1.MachineControlPlaneLabelName: "",
 					},
-					Finalizers:        []string{clusterv1.MachineFinalizer},
+					Finalizers:        []string{clusterv1.MachineFinalizer, metav1.FinalizerDeleteDependents},
 					DeletionTimestamp: &metav1.Time{Time: time.Now().UTC()},
 				},
 				Spec: clusterv1.MachineSpec{
@@ -874,7 +873,7 @@ func TestIsDeleteNodeAllowed(t *testing.T) {
 					Labels: map[string]string{
 						clusterv1.ClusterLabelName: "test",
 					},
-					Finalizers: []string{clusterv1.MachineFinalizer},
+					Finalizers: []string{clusterv1.MachineFinalizer, metav1.FinalizerDeleteDependents},
 				},
 				Spec: clusterv1.MachineSpec{
 					ClusterName:       "test-cluster",
@@ -912,7 +911,7 @@ func TestIsDeleteNodeAllowed(t *testing.T) {
 					Labels: map[string]string{
 						clusterv1.ClusterLabelName: "test",
 					},
-					Finalizers: []string{clusterv1.MachineFinalizer},
+					Finalizers: []string{clusterv1.MachineFinalizer, metav1.FinalizerDeleteDependents},
 				},
 				Spec: clusterv1.MachineSpec{
 					ClusterName:       "test-cluster",
@@ -932,7 +931,7 @@ func TestIsDeleteNodeAllowed(t *testing.T) {
 					Labels: map[string]string{
 						clusterv1.ClusterLabelName: "test",
 					},
-					Finalizers: []string{clusterv1.MachineFinalizer},
+					Finalizers: []string{clusterv1.MachineFinalizer, metav1.FinalizerDeleteDependents},
 				},
 				Spec: clusterv1.MachineSpec{
 					ClusterName:       "test-cluster",
diff --git a/controlplane/kubeadm/controllers/controller_test.go b/controlplane/kubeadm/controllers/controller_test.go
index 18669f08d..f5423e8ff 100644
--- a/controlplane/kubeadm/controllers/controller_test.go
+++ b/controlplane/kubeadm/controllers/controller_test.go
@@ -525,15 +525,6 @@ kubernetesVersion: metav1.16.1`,
 				Status: internal.ClusterStatus{},
 			},
 		},
-		managementClusterUncached: &fakeManagementCluster{
-			Management: &internal.Management{Client: fakeClient},
-			Workload: fakeWorkloadCluster{
-				Workload: &internal.Workload{
-					Client: fakeClient,
-				},
-				Status: internal.ClusterStatus{},
-			},
-		},
 	}
 
 	result, err := r.Reconcile(ctrl.Request{NamespacedName: util.ObjectKey(kcp)})
diff --git a/controlplane/kubeadm/controllers/scale_test.go b/controlplane/kubeadm/controllers/scale_test.go
index 72fe8688d..088d37167 100644
--- a/controlplane/kubeadm/controllers/scale_test.go
+++ b/controlplane/kubeadm/controllers/scale_test.go
@@ -50,10 +50,6 @@ func TestKubeadmControlPlaneReconciler_initializeControlPlane(t *testing.T) {
 		Client:   fakeClient,
 		Log:      log.Log,
 		recorder: record.NewFakeRecorder(32),
-		managementClusterUncached: &fakeManagementCluster{
-			Management: &internal.Management{Client: fakeClient},
-			Workload:   fakeWorkloadCluster{},
-		},
 	}
 	controlPlane := &internal.ControlPlane{
 		Cluster: cluster,
@@ -105,11 +101,10 @@ func TestKubeadmControlPlaneReconciler_scaleUpControlPlane(t *testing.T) {
 		fakeClient := newFakeClient(g, initObjs...)
 
 		r := &KubeadmControlPlaneReconciler{
-			Client:                    fakeClient,
-			managementCluster:         fmc,
-			managementClusterUncached: fmc,
-			Log:                       log.Log,
-			recorder:                  record.NewFakeRecorder(32),
+			Client:            fakeClient,
+			managementCluster: fmc,
+			Log:               log.Log,
+			recorder:          record.NewFakeRecorder(32),
 		}
 		controlPlane := &internal.ControlPlane{
 			KCP:      kcp,
@@ -161,11 +156,10 @@ func TestKubeadmControlPlaneReconciler_scaleUpControlPlane(t *testing.T) {
 			}
 
 			r := &KubeadmControlPlaneReconciler{
-				Client:                    fakeClient,
-				managementCluster:         fmc,
-				managementClusterUncached: fmc,
-				Log:                       log.Log,
-				recorder:                  record.NewFakeRecorder(32),
+				Client:            fakeClient,
+				managementCluster: fmc,
+				Log:               log.Log,
+				recorder:          record.NewFakeRecorder(32),
 			}
 			controlPlane := &internal.ControlPlane{
 				KCP:      kcp,
diff --git a/controlplane/kubeadm/controllers/upgrade_test.go b/controlplane/kubeadm/controllers/upgrade_test.go
index 7dbd859c0..5136054e5 100644
--- a/controlplane/kubeadm/controllers/upgrade_test.go
+++ b/controlplane/kubeadm/controllers/upgrade_test.go
@@ -53,12 +53,6 @@ func TestKubeadmControlPlaneReconciler_upgradeControlPlane(t *testing.T) {
 			ControlPlaneHealthy: true,
 			EtcdHealthy:         true,
 		},
-		managementClusterUncached: &fakeManagementCluster{
-			Management:          &internal.Management{Client: fakeClient},
-			Workload:            fakeWorkloadCluster{Status: internal.ClusterStatus{Nodes: 1}},
-			ControlPlaneHealthy: true,
-			EtcdHealthy:         true,
-		},
 	}
 	controlPlane := &internal.ControlPlane{
 		KCP:      kcp,
diff --git a/exp/controllers/machinepool_controller_test.go b/exp/controllers/machinepool_controller_test.go
index 95bfe45a4..26ed9bd16 100644
--- a/exp/controllers/machinepool_controller_test.go
+++ b/exp/controllers/machinepool_controller_test.go
@@ -17,6 +17,7 @@ limitations under the License.
 package controllers
 
 import (
+	"sigs.k8s.io/cluster-api/util"
 	"testing"
 
 	. "github.com/onsi/gomega"
@@ -29,7 +30,6 @@ import (
 	"k8s.io/utils/pointer"
 	clusterv1 "sigs.k8s.io/cluster-api/api/v1alpha3"
 	expv1 "sigs.k8s.io/cluster-api/exp/api/v1alpha3"
-	"sigs.k8s.io/cluster-api/util"
 	"sigs.k8s.io/controller-runtime/pkg/client"
 	"sigs.k8s.io/controller-runtime/pkg/client/fake"
 	"sigs.k8s.io/controller-runtime/pkg/log"
@@ -304,7 +304,7 @@ func TestReconcileMachinePoolRequest(t *testing.T) {
 				ObjectMeta: metav1.ObjectMeta{
 					Name:       "created",
 					Namespace:  "default",
-					Finalizers: []string{expv1.MachinePoolFinalizer},
+					Finalizers: []string{expv1.MachinePoolFinalizer, metav1.FinalizerDeleteDependents},
 				},
 				Spec: expv1.MachinePoolSpec{
 					ClusterName:    "test-cluster",
@@ -340,7 +340,7 @@ func TestReconcileMachinePoolRequest(t *testing.T) {
 				ObjectMeta: metav1.ObjectMeta{
 					Name:       "updated",
 					Namespace:  "default",
-					Finalizers: []string{expv1.MachinePoolFinalizer},
+					Finalizers: []string{expv1.MachinePoolFinalizer, metav1.FinalizerDeleteDependents},
 				},
 				Spec: expv1.MachinePoolSpec{
 					ClusterName:    "test-cluster",
@@ -378,7 +378,7 @@ func TestReconcileMachinePoolRequest(t *testing.T) {
 					Labels: map[string]string{
 						clusterv1.MachineControlPlaneLabelName: "",
 					},
-					Finalizers:        []string{expv1.MachinePoolFinalizer},
+					Finalizers:        []string{expv1.MachinePoolFinalizer, metav1.FinalizerDeleteDependents},
 					DeletionTimestamp: &time,
 				},
 				Spec: expv1.MachinePoolSpec{
@@ -574,7 +574,7 @@ func TestRemoveMachinePoolFinalizerAfterDeleteReconcile(t *testing.T) {
 		ObjectMeta: metav1.ObjectMeta{
 			Name:              "delete123",
 			Namespace:         "default",
-			Finalizers:        []string{expv1.MachinePoolFinalizer},
+			Finalizers:        []string{expv1.MachinePoolFinalizer, metav1.FinalizerDeleteDependents},
 			DeletionTimestamp: &dt,
 		},
 		Spec: expv1.MachinePoolSpec{
@@ -601,7 +601,6 @@ func TestRemoveMachinePoolFinalizerAfterDeleteReconcile(t *testing.T) {
 	_, err := mr.Reconcile(reconcile.Request{NamespacedName: key})
 	g.Expect(err).ToNot(HaveOccurred())
 
-	var actual expv1.MachinePool
-	g.Expect(mr.Client.Get(ctx, key, &actual)).To(Succeed())
-	g.Expect(actual.ObjectMeta.Finalizers).To(BeEmpty())
+	g.Expect(mr.Client.Get(ctx, key, m)).To(Succeed())
+	g.Expect(m.ObjectMeta.Finalizers).To(Equal([]string{metav1.FinalizerDeleteDependents}))
 }
diff --git a/test/e2e/common.go b/test/e2e/common.go
index 7104b2a8c..baad2876c 100644
--- a/test/e2e/common.go
+++ b/test/e2e/common.go
@@ -20,15 +20,28 @@ import (
 	"context"
 	"fmt"
 	"path/filepath"
+	"strings"
 
 	. "github.com/onsi/ginkgo"
 
+	"github.com/blang/semver"
+	"github.com/onsi/gomega/types"
 	corev1 "k8s.io/api/core/v1"
 	clusterv1 "sigs.k8s.io/cluster-api/api/v1alpha3"
 	"sigs.k8s.io/cluster-api/test/framework"
 	"sigs.k8s.io/cluster-api/util"
 )
 
+// Test suite constants for e2e config variables
+const (
+	KubernetesVersion            = "KUBERNETES_VERSION"
+	CNIPath                      = "CNI"
+	KubernetesVersionUpgradeFrom = "KUBERNETES_VERSION_UPGRADE_FROM"
+	KubernetesVersionUpgradeTo   = "KUBERNETES_VERSION_UPGRADE_TO"
+	EtcdVersionUpgradeTo         = "ETCD_VERSION_UPGRADE_TO"
+	CoreDNSVersionUpgradeTo      = "COREDNS_VERSION_UPGRADE_TO"
+)
+
 func Byf(format string, a ...interface{}) {
 	By(fmt.Sprintf(format, a...))
 }
@@ -72,3 +85,25 @@ func dumpSpecResourcesAndCleanup(ctx context.Context, specName string, clusterPr
 	}
 	cancelWatches()
 }
+
+// HaveValidVersion succeeds if version is a valid semver version
+func HaveValidVersion(version string) types.GomegaMatcher {
+	return &validVersionMatcher{version: version}
+}
+
+type validVersionMatcher struct{ version string }
+
+func (m *validVersionMatcher) Match(actual interface{}) (success bool, err error) {
+	if _, err := semver.Parse(strings.TrimPrefix(strings.TrimSpace(m.version), "v")); err != nil {
+		return false, err
+	}
+	return true, nil
+}
+
+func (m *validVersionMatcher) FailureMessage(actual interface{}) (message string) {
+	return fmt.Sprintf("Expected\n%s\n%s", m.version, " to be a valid version ")
+}
+
+func (m *validVersionMatcher) NegatedFailureMessage(actual interface{}) (message string) {
+	return fmt.Sprintf("Expected\n%s\n%s", m.version, " not to be a valid version ")
+}
diff --git a/test/e2e/config/docker-ci.yaml b/test/e2e/config/docker-ci.yaml
index cd5528189..6b2ce86ee 100644
--- a/test/e2e/config/docker-ci.yaml
+++ b/test/e2e/config/docker-ci.yaml
@@ -62,10 +62,10 @@ providers:
 
 variables:
   KUBERNETES_VERSION: "v1.18.2"
-  ETCD_VERSION_CURRENT: "3.4.3-0"
-  COREDNS_VERSION_CURRENT: "1.6.7"
+  ETCD_VERSION_UPGRADE_TO: "3.4.3-0"
+  COREDNS_VERSION_UPGRADE_TO: "1.6.7"
+  KUBERNETES_VERSION_UPGRADE_TO: "v1.18.2"
   KUBERNETES_VERSION_UPGRADE_FROM: "v1.17.2"
-  KUBERNETES_VERSION_MINIMUM_CAPI_SUPPORTED: "v1.16.4"
   DOCKER_SERVICE_DOMAIN: "cluster.local"
   DOCKER_SERVICE_CIDRS: "10.128.0.0/12"
   # IMPORTANT! This values should match the one used by the CNI provider
@@ -79,4 +79,4 @@ intervals:
   default/wait-worker-nodes: ["5m", "10s"]
   default/wait-delete-cluster: ["3m", "10s"]
   default/wait-machine-upgrade: ["20m", "10s"]
-
+  default/wait-machine-remediation: ["3m", "10s"]
diff --git a/test/e2e/config/docker-dev.yaml b/test/e2e/config/docker-dev.yaml
index 9c5849873..56159d633 100644
--- a/test/e2e/config/docker-dev.yaml
+++ b/test/e2e/config/docker-dev.yaml
@@ -92,10 +92,10 @@ providers:
 
 variables:
   KUBERNETES_VERSION: "v1.18.2"
-  ETCD_VERSION_CURRENT: "3.4.3-0"
-  COREDNS_VERSION_CURRENT: "1.6.7"
+  ETCD_VERSION_UPGRADE_TO: "3.4.3-0"
+  COREDNS_VERSION_UPGRADE_TO: "1.6.7"
+  KUBERNETES_VERSION_UPGRADE_TO: "v1.18.2"
   KUBERNETES_VERSION_UPGRADE_FROM: "v1.17.2"
-  KUBERNETES_VERSION_MINIMUM_CAPI_SUPPORTED: "v1.16.4"
   DOCKER_SERVICE_DOMAIN: "cluster.local"
   DOCKER_SERVICE_CIDRS: "10.128.0.0/12"
   # IMPORTANT! This values should match the one used by the CNI provider
@@ -110,3 +110,4 @@ intervals:
   default/wait-worker-nodes: ["3m", "10s"]
   default/wait-delete-cluster: ["3m", "10s"]
   default/wait-machine-upgrade: ["15m", "10s"]
+  default/wait-machine-remediation: ["3m", "10s"]
\ No newline at end of file
diff --git a/test/e2e/data/infrastructure-aws/cluster-template.yaml b/test/e2e/data/infrastructure-aws/cluster-template.yaml
deleted file mode 100644
index 0bafd4746..000000000
--- a/test/e2e/data/infrastructure-aws/cluster-template.yaml
+++ /dev/null
@@ -1,113 +0,0 @@
----
-apiVersion: cluster.x-k8s.io/v1alpha3
-kind: Cluster
-metadata:
-  name: "${CLUSTER_NAME}"
-spec:
-  clusterNetwork:
-    pods:
-      cidrBlocks: ["192.168.0.0/16"]
-  infrastructureRef:
-    apiVersion: infrastructure.cluster.x-k8s.io/v1alpha3
-    kind: AWSCluster
-    name: "${CLUSTER_NAME}"
-  controlPlaneRef:
-    kind: KubeadmControlPlane
-    apiVersion: controlplane.cluster.x-k8s.io/v1alpha3
-    name: "${CLUSTER_NAME}-control-plane"
----
-apiVersion: infrastructure.cluster.x-k8s.io/v1alpha3
-kind: AWSCluster
-metadata:
-  name: "${CLUSTER_NAME}"
-spec:
-  region: "${AWS_REGION}"
-  sshKeyName: "${AWS_SSH_KEY_NAME}"
----
-kind: KubeadmControlPlane
-apiVersion: controlplane.cluster.x-k8s.io/v1alpha3
-metadata:
-  name: "${CLUSTER_NAME}-control-plane"
-spec:
-  replicas: ${CONTROL_PLANE_MACHINE_COUNT}
-  infrastructureTemplate:
-    kind: AWSMachineTemplate
-    apiVersion: infrastructure.cluster.x-k8s.io/v1alpha3
-    name: "${CLUSTER_NAME}-control-plane"
-  kubeadmConfigSpec:
-    initConfiguration:
-      nodeRegistration:
-        name: '{{ ds.meta_data.local_hostname }}'
-        kubeletExtraArgs:
-          cloud-provider: aws
-    clusterConfiguration:
-      apiServer:
-        extraArgs:
-          cloud-provider: aws
-      controllerManager:
-        extraArgs:
-          cloud-provider: aws
-    joinConfiguration:
-      nodeRegistration:
-        name: '{{ ds.meta_data.local_hostname }}'
-        kubeletExtraArgs:
-          cloud-provider: aws
-  version: "${KUBERNETES_VERSION}"
----
-kind: AWSMachineTemplate
-apiVersion: infrastructure.cluster.x-k8s.io/v1alpha3
-metadata:
-  name: "${CLUSTER_NAME}-control-plane"
-spec:
-  template:
-    spec:
-      instanceType: "${AWS_CONTROL_PLANE_MACHINE_TYPE}"
-      iamInstanceProfile: "control-plane.cluster-api-provider-aws.sigs.k8s.io"
-      sshKeyName: "${AWS_SSH_KEY_NAME}"
----
-apiVersion: cluster.x-k8s.io/v1alpha3
-kind: MachineDeployment
-metadata:
-  name: "${CLUSTER_NAME}-md-0"
-spec:
-  clusterName: "${CLUSTER_NAME}"
-  replicas: ${WORKER_MACHINE_COUNT}
-  selector:
-    matchLabels:
-  template:
-    spec:
-      clusterName: "${CLUSTER_NAME}"
-      version: "${KUBERNETES_VERSION}"
-      bootstrap:
-        configRef:
-          name: "${CLUSTER_NAME}-md-0"
-          apiVersion: bootstrap.cluster.x-k8s.io/v1alpha3
-          kind: KubeadmConfigTemplate
-      infrastructureRef:
-        name: "${CLUSTER_NAME}-md-0"
-        apiVersion: infrastructure.cluster.x-k8s.io/v1alpha3
-        kind: AWSMachineTemplate
----
-apiVersion: infrastructure.cluster.x-k8s.io/v1alpha3
-kind: AWSMachineTemplate
-metadata:
-  name: "${CLUSTER_NAME}-md-0"
-spec:
-  template:
-    spec:
-      instanceType: "${AWS_NODE_MACHINE_TYPE}"
-      iamInstanceProfile: "nodes.cluster-api-provider-aws.sigs.k8s.io"
-      sshKeyName: "${AWS_SSH_KEY_NAME}"
----
-apiVersion: bootstrap.cluster.x-k8s.io/v1alpha3
-kind: KubeadmConfigTemplate
-metadata:
-  name: "${CLUSTER_NAME}-md-0"
-spec:
-  template:
-    spec:
-      joinConfiguration:
-        nodeRegistration:
-          name: '{{ ds.meta_data.local_hostname }}'
-          kubeletExtraArgs:
-            cloud-provider: aws
diff --git a/test/e2e/kcp_upgrade.go b/test/e2e/kcp_upgrade.go
index a82b4d681..4e4880e74 100644
--- a/test/e2e/kcp_upgrade.go
+++ b/test/e2e/kcp_upgrade.go
@@ -34,12 +34,6 @@ import (
 	"sigs.k8s.io/cluster-api/util"
 )
 
-const (
-	PreviousKubernetesVersion = "KUBERNETES_VERSION_UPGRADE_FROM"
-	EtcdCurrentVersion        = "ETCD_VERSION_CURRENT"
-	CoreDNSCurrentVersion     = "COREDNS_VERSION_CURRENT"
-)
-
 // KCPUpgradeSpecInput is the input for KCPUpgradeSpec.
 type KCPUpgradeSpecInput struct {
 	E2EConfig             *clusterctl.E2EConfig
@@ -67,9 +61,11 @@ func KCPUpgradeSpec(ctx context.Context, inputGetter func() KCPUpgradeSpecInput)
 		Expect(input.ClusterctlConfigPath).To(BeAnExistingFile(), "Invalid argument. input.ClusterctlConfigPath must be an existing file when calling %s spec", specName)
 		Expect(input.BootstrapClusterProxy).ToNot(BeNil(), "Invalid argument. input.BootstrapClusterProxy can't be nil when calling %s spec", specName)
 		Expect(os.MkdirAll(input.ArtifactFolder, 0755)).To(Succeed(), "Invalid argument. input.ArtifactFolder can't be created for %s spec", specName)
-		Expect(input.E2EConfig.Variables).To(HaveKey(PreviousKubernetesVersion))
-		Expect(input.E2EConfig.Variables).To(HaveKey(EtcdCurrentVersion))
-		Expect(input.E2EConfig.Variables).To(HaveKey(CoreDNSCurrentVersion))
+		Expect(input.E2EConfig.Variables).To(HaveKey(KubernetesVersionUpgradeTo))
+		Expect(input.E2EConfig.Variables).To(HaveKey(CNIPath))
+		Expect(input.E2EConfig.Variables).To(HaveKey(KubernetesVersionUpgradeFrom))
+		Expect(input.E2EConfig.Variables).To(HaveKey(EtcdVersionUpgradeTo))
+		Expect(input.E2EConfig.Variables).To(HaveKey(CoreDNSVersionUpgradeTo))
 
 		// Setup a Namespace where to host objects for this spec and create a watcher for the namespace events.
 		namespace, cancelWatches = setupSpecNamespace(ctx, specName, input.BootstrapClusterProxy, input.ArtifactFolder)
@@ -78,9 +74,6 @@ func KCPUpgradeSpec(ctx context.Context, inputGetter func() KCPUpgradeSpecInput)
 	It("Should successfully upgrade Kubernetes, DNS, kube-proxy, and etcd in a single control plane cluster", func() {
 
 		By("Creating a workload cluster")
-		Expect(input.E2EConfig.Variables).To(HaveKey(clusterctl.KubernetesVersion))
-		Expect(input.E2EConfig.Variables).To(HaveKey(clusterctl.CNIPath))
-
 		cluster, controlPlane, _ = clusterctl.ApplyClusterTemplateAndWait(ctx, clusterctl.ApplyClusterTemplateAndWaitInput{
 			ClusterProxy: input.BootstrapClusterProxy,
 			ConfigCluster: clusterctl.ConfigClusterInput{
@@ -91,11 +84,11 @@ func KCPUpgradeSpec(ctx context.Context, inputGetter func() KCPUpgradeSpecInput)
 				Flavor:                   clusterctl.DefaultFlavor,
 				Namespace:                namespace.Name,
 				ClusterName:              fmt.Sprintf("cluster-%s", util.RandomString(6)),
-				KubernetesVersion:        input.GetPreviousKubernetesVersion(),
+				KubernetesVersion:        input.E2EConfig.GetVariable(KubernetesVersionUpgradeFrom),
 				ControlPlaneMachineCount: pointer.Int64Ptr(1),
 				WorkerMachineCount:       pointer.Int64Ptr(1),
 			},
-			CNIManifestPath:              input.E2EConfig.GetCNIPath(),
+			CNIManifestPath:              input.E2EConfig.GetVariable(CNIPath),
 			WaitForClusterIntervals:      input.E2EConfig.GetIntervals(specName, "wait-cluster"),
 			WaitForControlPlaneIntervals: input.E2EConfig.GetIntervals(specName, "wait-control-plane"),
 			WaitForMachineDeployments:    input.E2EConfig.GetIntervals(specName, "wait-worker-nodes"),
@@ -103,13 +96,12 @@ func KCPUpgradeSpec(ctx context.Context, inputGetter func() KCPUpgradeSpecInput)
 
 		By("Upgrading Kubernetes, DNS, kube-proxy, and etcd versions")
 		framework.UpgradeControlPlaneAndWaitForUpgrade(ctx, framework.UpgradeControlPlaneAndWaitForUpgradeInput{
-			ClusterProxy: input.BootstrapClusterProxy,
-			Cluster:      cluster,
-			ControlPlane: controlPlane,
-			//Valid image tags for v1.17.2
-			EtcdImageTag:                "3.4.3-0",
-			DNSImageTag:                 "1.6.7",
-			KubernetesUpgradeVersion:    input.E2EConfig.GetKubernetesVersion(),
+			ClusterProxy:                input.BootstrapClusterProxy,
+			Cluster:                     cluster,
+			ControlPlane:                controlPlane,
+			EtcdImageTag:                input.E2EConfig.GetVariable(EtcdVersionUpgradeTo),
+			DNSImageTag:                 input.E2EConfig.GetVariable(CoreDNSVersionUpgradeTo),
+			KubernetesUpgradeVersion:    input.E2EConfig.GetVariable(KubernetesVersionUpgradeTo),
 			WaitForMachinesToBeUpgraded: input.E2EConfig.GetIntervals(specName, "wait-machine-upgrade"),
 			WaitForDNSUpgrade:           input.E2EConfig.GetIntervals(specName, "wait-machine-upgrade"),
 			WaitForEtcdUpgrade:          input.E2EConfig.GetIntervals(specName, "wait-machine-upgrade"),
@@ -121,8 +113,6 @@ func KCPUpgradeSpec(ctx context.Context, inputGetter func() KCPUpgradeSpecInput)
 	It("Should successfully upgrade Kubernetes, DNS, kube-proxy, and etcd in a HA cluster", func() {
 
 		By("Creating a workload cluster")
-		Expect(input.E2EConfig.Variables).To(HaveKey(clusterctl.KubernetesVersion))
-		Expect(input.E2EConfig.Variables).To(HaveKey(clusterctl.CNIPath))
 
 		cluster, controlPlane, _ = clusterctl.ApplyClusterTemplateAndWait(ctx, clusterctl.ApplyClusterTemplateAndWaitInput{
 			ClusterProxy: input.BootstrapClusterProxy,
@@ -134,24 +124,24 @@ func KCPUpgradeSpec(ctx context.Context, inputGetter func() KCPUpgradeSpecInput)
 				Flavor:                   clusterctl.DefaultFlavor,
 				Namespace:                namespace.Name,
 				ClusterName:              fmt.Sprintf("cluster-%s", util.RandomString(6)),
-				KubernetesVersion:        input.GetPreviousKubernetesVersion(),
+				KubernetesVersion:        input.E2EConfig.GetVariable(KubernetesVersionUpgradeFrom),
 				ControlPlaneMachineCount: pointer.Int64Ptr(3),
 				WorkerMachineCount:       pointer.Int64Ptr(1),
 			},
-			CNIManifestPath:              input.E2EConfig.GetCNIPath(),
+			CNIManifestPath:              input.E2EConfig.GetVariable(CNIPath),
 			WaitForClusterIntervals:      input.E2EConfig.GetIntervals(specName, "wait-cluster"),
 			WaitForControlPlaneIntervals: input.E2EConfig.GetIntervals(specName, "wait-control-plane"),
 			WaitForMachineDeployments:    input.E2EConfig.GetIntervals(specName, "wait-worker-nodes"),
 		})
 
-		By("Upgrading Kubernetes")
+		By("Upgrading Kubernetes, DNS, kube-proxy, and etcd versions")
 		framework.UpgradeControlPlaneAndWaitForUpgrade(ctx, framework.UpgradeControlPlaneAndWaitForUpgradeInput{
 			ClusterProxy:                input.BootstrapClusterProxy,
 			Cluster:                     cluster,
 			ControlPlane:                controlPlane,
-			EtcdImageTag:                input.GetEtcdCurrentVersion(),
-			DNSImageTag:                 input.GetCoreDNSCurrentVersion(),
-			KubernetesUpgradeVersion:    input.E2EConfig.GetKubernetesVersion(),
+			EtcdImageTag:                input.E2EConfig.GetVariable(EtcdVersionUpgradeTo),
+			DNSImageTag:                 input.E2EConfig.GetVariable(CoreDNSVersionUpgradeTo),
+			KubernetesUpgradeVersion:    input.E2EConfig.GetVariable(KubernetesVersionUpgradeTo),
 			WaitForMachinesToBeUpgraded: input.E2EConfig.GetIntervals(specName, "wait-machine-upgrade"),
 			WaitForDNSUpgrade:           input.E2EConfig.GetIntervals(specName, "wait-machine-upgrade"),
 			WaitForEtcdUpgrade:          input.E2EConfig.GetIntervals(specName, "wait-machine-upgrade"),
@@ -165,18 +155,3 @@ func KCPUpgradeSpec(ctx context.Context, inputGetter func() KCPUpgradeSpecInput)
 		dumpSpecResourcesAndCleanup(ctx, specName, input.BootstrapClusterProxy, input.ArtifactFolder, namespace, cancelWatches, cluster, input.E2EConfig.GetIntervals, input.SkipCleanup)
 	})
 }
-
-// GetPreviousKubernetesVersion returns the previous kubernetes version to test an upgrade from
-func (k KCPUpgradeSpecInput) GetPreviousKubernetesVersion() string {
-	return k.E2EConfig.Variables[PreviousKubernetesVersion]
-}
-
-// GetEtcdCurrentVersion returns the version of etcd to upgrade to
-func (k KCPUpgradeSpecInput) GetEtcdCurrentVersion() string {
-	return k.E2EConfig.Variables[EtcdCurrentVersion]
-}
-
-// GetCoreDNSUpgradeVersion returns the version of etcd to upgrade to
-func (k KCPUpgradeSpecInput) GetCoreDNSCurrentVersion() string {
-	return k.E2EConfig.Variables[CoreDNSCurrentVersion]
-}
diff --git a/test/e2e/md_upgrades.go b/test/e2e/md_upgrades.go
new file mode 100644
index 000000000..2f772c4aa
--- /dev/null
+++ b/test/e2e/md_upgrades.go
@@ -0,0 +1,120 @@
+/*
+Copyright 2020 The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package e2e
+
+import (
+	"context"
+	"fmt"
+	"os"
+	"path/filepath"
+
+	. "github.com/onsi/ginkgo"
+	. "github.com/onsi/gomega"
+
+	corev1 "k8s.io/api/core/v1"
+	"k8s.io/utils/pointer"
+	clusterv1 "sigs.k8s.io/cluster-api/api/v1alpha3"
+	"sigs.k8s.io/cluster-api/test/framework"
+	"sigs.k8s.io/cluster-api/test/framework/clusterctl"
+	"sigs.k8s.io/cluster-api/util"
+)
+
+// MachineDeploymentUpgradesSpecInput is the input for MachineDeploymentUpgradesSpec.
+type MachineDeploymentUpgradesSpecInput struct {
+	E2EConfig             *clusterctl.E2EConfig
+	ClusterctlConfigPath  string
+	BootstrapClusterProxy framework.ClusterProxy
+	ArtifactFolder        string
+	SkipCleanup           bool
+}
+
+// MachineDeploymentUpgradesSpec implements a test that verifies that MachineDeployment upgrades are successful.
+func MachineDeploymentUpgradesSpec(ctx context.Context, inputGetter func() MachineDeploymentUpgradesSpecInput) {
+	var (
+		specName      = "md-upgrades"
+		input         MachineDeploymentUpgradesSpecInput
+		namespace     *corev1.Namespace
+		cancelWatches context.CancelFunc
+		cluster       *clusterv1.Cluster
+	)
+
+	BeforeEach(func() {
+		Expect(ctx).NotTo(BeNil(), "ctx is required for %s spec", specName)
+		input = inputGetter()
+		Expect(input.E2EConfig).ToNot(BeNil(), "Invalid argument. input.E2EConfig can't be nil when calling %s spec", specName)
+		Expect(input.ClusterctlConfigPath).To(BeAnExistingFile(), "Invalid argument. input.ClusterctlConfigPath must be an existing file when calling %s spec", specName)
+		Expect(input.BootstrapClusterProxy).ToNot(BeNil(), "Invalid argument. input.BootstrapClusterProxy can't be nil when calling %s spec", specName)
+		Expect(os.MkdirAll(input.ArtifactFolder, 0755)).To(Succeed(), "Invalid argument. input.ArtifactFolder can't be created for %s spec", specName)
+		Expect(input.E2EConfig.Variables).To(HaveKey(KubernetesVersion))
+		Expect(input.E2EConfig.Variables).To(HaveValidVersion(input.E2EConfig.GetVariable(KubernetesVersion)))
+		Expect(input.E2EConfig.Variables).To(HaveKey(KubernetesVersionUpgradeFrom))
+		Expect(input.E2EConfig.Variables).To(HaveValidVersion(input.E2EConfig.GetVariable(KubernetesVersionUpgradeFrom)))
+		Expect(input.E2EConfig.Variables).To(HaveKey(CNIPath))
+
+		// Setup a Namespace where to host objects for this spec and create a watcher for the namespace events.
+		namespace, cancelWatches = setupSpecNamespace(ctx, specName, input.BootstrapClusterProxy, input.ArtifactFolder)
+	})
+
+	It("Should successfully upgrade Machines upon changes in relevant MachineDeployment fields", func() {
+
+		By("Creating a workload cluster")
+
+		var mds []*clusterv1.MachineDeployment
+		cluster, _, mds = clusterctl.ApplyClusterTemplateAndWait(ctx, clusterctl.ApplyClusterTemplateAndWaitInput{
+			ClusterProxy: input.BootstrapClusterProxy,
+			ConfigCluster: clusterctl.ConfigClusterInput{
+				LogFolder:                filepath.Join(input.ArtifactFolder, "clusters", input.BootstrapClusterProxy.GetName()),
+				ClusterctlConfigPath:     input.ClusterctlConfigPath,
+				KubeconfigPath:           input.BootstrapClusterProxy.GetKubeconfigPath(),
+				InfrastructureProvider:   clusterctl.DefaultInfrastructureProvider,
+				Flavor:                   clusterctl.DefaultFlavor,
+				Namespace:                namespace.Name,
+				ClusterName:              fmt.Sprintf("cluster-%s", util.RandomString(6)),
+				KubernetesVersion:        input.E2EConfig.GetVariable(KubernetesVersionUpgradeFrom),
+				ControlPlaneMachineCount: pointer.Int64Ptr(1),
+				WorkerMachineCount:       pointer.Int64Ptr(1),
+			},
+			CNIManifestPath:              input.E2EConfig.GetVariable(CNIPath),
+			WaitForClusterIntervals:      input.E2EConfig.GetIntervals(specName, "wait-cluster"),
+			WaitForControlPlaneIntervals: input.E2EConfig.GetIntervals(specName, "wait-control-plane"),
+			WaitForMachineDeployments:    input.E2EConfig.GetIntervals(specName, "wait-worker-nodes"),
+		})
+
+		By("Upgrading MachineDeployment's Kubernetes version to a valid version")
+		framework.UpgradeMachineDeploymentsAndWait(context.TODO(), framework.UpgradeMachineDeploymentsAndWaitInput{
+			ClusterProxy:                input.BootstrapClusterProxy,
+			Cluster:                     cluster,
+			UpgradeVersion:              input.E2EConfig.GetVariable(KubernetesVersion),
+			WaitForMachinesToBeUpgraded: input.E2EConfig.GetIntervals(specName, "wait-machine-upgrade"),
+			MachineDeployments:          mds,
+		})
+
+		By("Upgrading MachineDeployment Infrastructure ref and wait for rolling upgrade")
+		framework.UpgradeMachineDeploymentInfrastructureRefAndWait(context.TODO(), framework.UpgradeMachineDeploymentInfrastructureRefAndWaitInput{
+			ClusterProxy:                input.BootstrapClusterProxy,
+			Cluster:                     cluster,
+			WaitForMachinesToBeUpgraded: input.E2EConfig.GetIntervals(specName, "wait-machine-upgrade"),
+			MachineDeployments:          mds,
+		})
+		By("PASSED!")
+	})
+
+	AfterEach(func() {
+		// Dumps all the resources in the spec namespace, then cleanups the cluster object and the spec namespace itself.
+		dumpSpecResourcesAndCleanup(ctx, specName, input.BootstrapClusterProxy, input.ArtifactFolder, namespace, cancelWatches, cluster, input.E2EConfig.GetIntervals, input.SkipCleanup)
+	})
+}
diff --git a/test/e2e/md_upgrades_test.go b/test/e2e/md_upgrades_test.go
new file mode 100644
index 000000000..e709ad106
--- /dev/null
+++ b/test/e2e/md_upgrades_test.go
@@ -0,0 +1,39 @@
+// +build e2e
+
+/*
+Copyright 2020 The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package e2e
+
+import (
+	"context"
+
+	. "github.com/onsi/ginkgo"
+)
+
+var _ = Describe("When testing MachineDeployment upgrades", func() {
+
+	MachineDeploymentUpgradesSpec(context.TODO(), func() MachineDeploymentUpgradesSpecInput {
+		return MachineDeploymentUpgradesSpecInput{
+			E2EConfig:             e2eConfig,
+			ClusterctlConfigPath:  clusterctlConfigPath,
+			BootstrapClusterProxy: bootstrapClusterProxy,
+			ArtifactFolder:        artifactFolder,
+			SkipCleanup:           skipCleanup,
+		}
+	})
+
+})
diff --git a/test/e2e/quick_start.go b/test/e2e/quick_start.go
index 75cc48828..a0b2959b0 100644
--- a/test/e2e/quick_start.go
+++ b/test/e2e/quick_start.go
@@ -62,6 +62,9 @@ func QuickStartSpec(ctx context.Context, inputGetter func() QuickStartSpecInput)
 		Expect(input.BootstrapClusterProxy).ToNot(BeNil(), "Invalid argument. input.BootstrapClusterProxy can't be nil when calling %s spec", specName)
 		Expect(os.MkdirAll(input.ArtifactFolder, 0755)).To(Succeed(), "Invalid argument. input.ArtifactFolder can't be created for %s spec", specName)
 
+		Expect(input.E2EConfig.Variables).To(HaveKey(KubernetesVersion))
+		Expect(input.E2EConfig.Variables).To(HaveKey(CNIPath))
+
 		// Setup a Namespace where to host objects for this spec and create a watcher for the namespace events.
 		namespace, cancelWatches = setupSpecNamespace(ctx, specName, input.BootstrapClusterProxy, input.ArtifactFolder)
 	})
@@ -69,8 +72,6 @@ func QuickStartSpec(ctx context.Context, inputGetter func() QuickStartSpecInput)
 	It("Should create a workload cluster", func() {
 
 		By("Creating a workload cluster")
-		Expect(input.E2EConfig.Variables).To(HaveKey(clusterctl.KubernetesVersion))
-		Expect(input.E2EConfig.Variables).To(HaveKey(clusterctl.CNIPath))
 
 		cluster, _, _ = clusterctl.ApplyClusterTemplateAndWait(ctx, clusterctl.ApplyClusterTemplateAndWaitInput{
 			ClusterProxy: input.BootstrapClusterProxy,
@@ -82,11 +83,11 @@ func QuickStartSpec(ctx context.Context, inputGetter func() QuickStartSpecInput)
 				Flavor:                   clusterctl.DefaultFlavor,
 				Namespace:                namespace.Name,
 				ClusterName:              fmt.Sprintf("cluster-%s", util.RandomString(6)),
-				KubernetesVersion:        input.E2EConfig.GetKubernetesVersion(),
+				KubernetesVersion:        input.E2EConfig.GetVariable(KubernetesVersion),
 				ControlPlaneMachineCount: pointer.Int64Ptr(1),
 				WorkerMachineCount:       pointer.Int64Ptr(1),
 			},
-			CNIManifestPath:              input.E2EConfig.GetCNIPath(),
+			CNIManifestPath:              input.E2EConfig.GetVariable(CNIPath),
 			WaitForClusterIntervals:      input.E2EConfig.GetIntervals(specName, "wait-cluster"),
 			WaitForControlPlaneIntervals: input.E2EConfig.GetIntervals(specName, "wait-control-plane"),
 			WaitForMachineDeployments:    input.E2EConfig.GetIntervals(specName, "wait-worker-nodes"),
diff --git a/test/e2e/self_hosted.go b/test/e2e/self_hosted.go
index 450a89754..8070de121 100644
--- a/test/e2e/self_hosted.go
+++ b/test/e2e/self_hosted.go
@@ -34,8 +34,6 @@ import (
 	"sigs.k8s.io/cluster-api/util"
 )
 
-const MinimumCAPISupportedKubernetesVersion = "KUBERNETES_VERSION_MINIMUM_CAPI_SUPPORTED"
-
 // SelfHostedSpecInput is the input for SelfHostedSpec.
 type SelfHostedSpecInput struct {
 	E2EConfig             *clusterctl.E2EConfig
@@ -67,17 +65,16 @@ func SelfHostedSpec(ctx context.Context, inputGetter func() SelfHostedSpecInput)
 		Expect(input.ClusterctlConfigPath).To(BeAnExistingFile(), "Invalid argument. input.ClusterctlConfigPath must be an existing file when calling %s spec", specName)
 		Expect(input.BootstrapClusterProxy).ToNot(BeNil(), "Invalid argument. input.BootstrapClusterProxy can't be nil when calling %s spec", specName)
 		Expect(os.MkdirAll(input.ArtifactFolder, 0755)).To(Succeed(), "Invalid argument. input.ArtifactFolder can't be created for %s spec", specName)
-		Expect(input.E2EConfig.Variables).To(HaveKey(MinimumCAPISupportedKubernetesVersion))
+		Expect(input.E2EConfig.Variables).To(HaveKey(CNIPath))
+		Expect(input.E2EConfig.Variables).To(HaveKey(KubernetesVersion))
 
 		// Setup a Namespace where to host objects for this spec and create a watcher for the namespace events.
 		namespace, cancelWatches = setupSpecNamespace(context.TODO(), specName, input.BootstrapClusterProxy, input.ArtifactFolder)
 	})
 
-	It("Should create a workload cluster", func() {
+	It("Should pivot the bootstrap cluster to a self-hosted cluster", func() {
 
 		By("Creating a workload cluster")
-		Expect(input.E2EConfig.Variables).To(HaveKey(clusterctl.KubernetesVersion))
-		Expect(input.E2EConfig.Variables).To(HaveKey(clusterctl.CNIPath))
 
 		cluster, _, _ = clusterctl.ApplyClusterTemplateAndWait(context.TODO(), clusterctl.ApplyClusterTemplateAndWaitInput{
 			ClusterProxy: input.BootstrapClusterProxy,
@@ -89,11 +86,11 @@ func SelfHostedSpec(ctx context.Context, inputGetter func() SelfHostedSpecInput)
 				Flavor:                   clusterctl.DefaultFlavor,
 				Namespace:                namespace.Name,
 				ClusterName:              fmt.Sprintf("cluster-%s", util.RandomString(6)),
-				KubernetesVersion:        input.GetMinimumCAPISupportedKubernetesVersion(),
+				KubernetesVersion:        input.E2EConfig.GetVariable(KubernetesVersion),
 				ControlPlaneMachineCount: pointer.Int64Ptr(1),
 				WorkerMachineCount:       pointer.Int64Ptr(1),
 			},
-			CNIManifestPath:              input.E2EConfig.GetCNIPath(),
+			CNIManifestPath:              input.E2EConfig.GetVariable(CNIPath),
 			WaitForClusterIntervals:      input.E2EConfig.GetIntervals(specName, "wait-cluster"),
 			WaitForControlPlaneIntervals: input.E2EConfig.GetIntervals(specName, "wait-control-plane"),
 			WaitForMachineDeployments:    input.E2EConfig.GetIntervals(specName, "wait-worker-nodes"),
@@ -193,7 +190,3 @@ func SelfHostedSpec(ctx context.Context, inputGetter func() SelfHostedSpecInput)
 		dumpSpecResourcesAndCleanup(ctx, specName, input.BootstrapClusterProxy, input.ArtifactFolder, namespace, cancelWatches, cluster, input.E2EConfig.GetIntervals, input.SkipCleanup)
 	})
 }
-
-func (k SelfHostedSpecInput) GetMinimumCAPISupportedKubernetesVersion() string {
-	return k.E2EConfig.Variables[MinimumCAPISupportedKubernetesVersion]
-}
diff --git a/test/framework/clusterctl/client.go b/test/framework/clusterctl/client.go
index 9b4eb216f..ce6ce60b0 100644
--- a/test/framework/clusterctl/client.go
+++ b/test/framework/clusterctl/client.go
@@ -19,6 +19,7 @@ package clusterctl
 import (
 	"context"
 	"fmt"
+	"os"
 	"strings"
 
 	. "github.com/onsi/ginkgo"
@@ -146,6 +147,12 @@ type MoveInput struct {
 
 // Move moves workload clusters.
 func Move(ctx context.Context, input MoveInput) {
+	Expect(ctx).NotTo(BeNil(), "ctx is required for Move")
+	Expect(input.ClusterctlConfigPath).To(BeAnExistingFile(), "Invalid argument. input.ClusterctlConfigPath must be an existing file when calling Move")
+	Expect(input.FromKubeconfigPath).To(BeAnExistingFile(), "Invalid argument. input.FromKubeconfigPath must be an existing file when calling Move")
+	Expect(input.ToKubeconfigPath).To(BeAnExistingFile(), "Invalid argument. input.ToKubeconfigPath must be an existing file when calling Move")
+	Expect(os.MkdirAll(input.LogFolder, 0755)).To(Succeed(), "Invalid argument. input.LogFolder can't be created for Move")
+
 	By("Moving workload clusters")
 
 	clusterctlClient, log := getClusterctlClientWithLogger(input.ClusterctlConfigPath, "clusterctl-move.log", input.LogFolder)
diff --git a/test/framework/clusterctl/e2e_config.go b/test/framework/clusterctl/e2e_config.go
index dd93b919f..7934ce939 100644
--- a/test/framework/clusterctl/e2e_config.go
+++ b/test/framework/clusterctl/e2e_config.go
@@ -40,12 +40,6 @@ import (
 
 // Provides access to the configuration for an e2e test.
 
-// Define constants for e2e config variables
-const (
-	KubernetesVersion = "KUBERNETES_VERSION"
-	CNIPath           = "CNI"
-)
-
 // LoadE2EConfigInput is the input for LoadE2EConfig.
 type LoadE2EConfigInput struct {
 	// ConfigPath for the e2e test.
@@ -192,8 +186,6 @@ func errEmptyArg(argName string) error {
 // - There should be one InfraProvider (pick your own).
 // - Image should have name and loadBehavior be one of [mustload, tryload].
 // - Intervals should be valid ginkgo intervals.
-// - KubernetesVersion is not nil and valid.
-// - CNIPath is not nil.
 func (c *E2EConfig) Validate() error {
 	// ManagementClusterName should not be empty.
 	if c.ManagementClusterName == "" {
@@ -232,20 +224,6 @@ func (c *E2EConfig) Validate() error {
 			}
 		}
 	}
-
-	// If KubernetesVersion is nil or not valid, return error.
-	k8sVersion := c.GetKubernetesVersion()
-	if k8sVersion == "" {
-		return errEmptyArg(fmt.Sprintf("Variables[%s]", KubernetesVersion))
-	} else if _, err := version.ParseSemantic(k8sVersion); err != nil {
-		return errInvalidArg("Variables[%s]=%q", KubernetesVersion, k8sVersion)
-	}
-
-	// If CniPath is nil, return error.
-	cniPath := c.GetCNIPath()
-	if cniPath == "" {
-		return errEmptyArg(fmt.Sprintf("Variables[%s]", CNIPath))
-	}
 	return nil
 }
 
@@ -394,9 +372,7 @@ func (c *E2EConfig) GetIntervals(spec, key string) []interface{} {
 // GetVariable returns a variable from the e2e config file.
 func (c *E2EConfig) GetVariable(varName string) string {
 	version, ok := c.Variables[varName]
-	if !ok {
-		return ""
-	}
+	Expect(ok).NotTo(BeFalse())
 	return version
 }
 
@@ -411,21 +387,3 @@ func (c *E2EConfig) GetInt64PtrVariable(varName string) *int64 {
 	Expect(err).NotTo(HaveOccurred())
 	return pointer.Int64Ptr(wCount)
 }
-
-// GetKubernetesVersion returns the kubernetes version provided in e2e config.
-func (c *E2EConfig) GetKubernetesVersion() string {
-	version, ok := c.Variables[KubernetesVersion]
-	if !ok {
-		return ""
-	}
-	return version
-}
-
-// GetCNIPath returns the CNI path provided in e2e config.
-func (c *E2EConfig) GetCNIPath() string {
-	path, ok := c.Variables[CNIPath]
-	if !ok {
-		return ""
-	}
-	return path
-}
diff --git a/test/framework/controlpane_helpers.go b/test/framework/controlpane_helpers.go
index 537c504cd..128589fd3 100644
--- a/test/framework/controlpane_helpers.go
+++ b/test/framework/controlpane_helpers.go
@@ -116,6 +116,10 @@ type WaitForOneKubeadmControlPlaneMachineToExistInput struct {
 
 // WaitForKubeadmControlPlaneMachineToExist will wait until all control plane machines have node refs.
 func WaitForOneKubeadmControlPlaneMachineToExist(ctx context.Context, input WaitForOneKubeadmControlPlaneMachineToExistInput, intervals ...interface{}) {
+	Expect(ctx).NotTo(BeNil(), "ctx is required for WaitForOneKubeadmControlPlaneMachineToExist")
+	Expect(input.Lister).ToNot(BeNil(), "Invalid argument. input.Getter can't be nil when calling WaitForOneKubeadmControlPlaneMachineToExist")
+	Expect(input.ControlPlane).ToNot(BeNil(), "Invalid argument. input.ControlPlane can't be nil when calling WaitForOneKubeadmControlPlaneMachineToExist")
+
 	By("waiting for one control plane node to exist")
 	inClustersNamespaceListOption := client.InNamespace(input.Cluster.Namespace)
 	// ControlPlane labels
@@ -205,31 +209,6 @@ func AssertControlPlaneFailureDomains(ctx context.Context, input AssertControlPl
 	Expect(failureDomainCounts).To(Equal(input.ExpectedFailureDomains))
 }
 
-type GetMachinesByClusterInput struct {
-	Lister      Lister
-	ClusterName string
-	Namespace   string
-}
-
-// GetControlPlaneMachinesByCluster returns the Machine objects for a cluster.
-// Important! this method relies on labels that are created by the CAPI controllers during the first reconciliation, so
-// it is necessary to ensure this is already happened before calling it.
-func GetControlPlaneMachinesByCluster(ctx context.Context, input GetMachinesByClusterInput) []clusterv1.Machine {
-	options := append(byClusterOptions(input.ClusterName, input.Namespace), controlPlaneMachineOptions()...)
-
-	machineList := &clusterv1.MachineList{}
-	Expect(input.Lister.List(ctx, machineList, options...)).To(Succeed(), "Failed to list MachineList object for Cluster %s/%s", input.Namespace, input.ClusterName)
-
-	return machineList.Items
-}
-
-// controlPlaneMachineOptions returns a set of ListOptions that allows to get all machine objects belonging to control plane.
-func controlPlaneMachineOptions() []client.ListOption {
-	return []client.ListOption{
-		client.HasLabels{clusterv1.MachineControlPlaneLabelName},
-	}
-}
-
 // DiscoveryAndWaitForControlPlaneInitializedInput is the input type for DiscoveryAndWaitForControlPlaneInitialized.
 type DiscoveryAndWaitForControlPlaneInitializedInput struct {
 	Lister  Lister
@@ -336,7 +315,7 @@ func UpgradeControlPlaneAndWaitForUpgrade(ctx context.Context, input UpgradeCont
 	Expect(patchHelper.Patch(ctx, input.ControlPlane)).To(Succeed())
 
 	fmt.Fprintf(GinkgoWriter, "Waiting for machines to have the upgraded kubernetes version\n")
-	WaitForMachinesToBeUpgraded(ctx, WaitForMachinesToBeUpgradedInput{
+	WaitForControlPlaneMachinesToBeUpgraded(ctx, WaitForControlPlaneMachinesToBeUpgradedInput{
 		Lister:                   mgmtClient,
 		Cluster:                  input.Cluster,
 		MachineCount:             int(*input.ControlPlane.Spec.Replicas),
@@ -366,3 +345,10 @@ func UpgradeControlPlaneAndWaitForUpgrade(ctx context.Context, input UpgradeCont
 		Condition:   EtcdImageTagCondition(input.EtcdImageTag, int(*input.ControlPlane.Spec.Replicas)),
 	}, input.WaitForEtcdUpgrade...)
 }
+
+// controlPlaneMachineOptions returns a set of ListOptions that allows to get all machine objects belonging to control plane.
+func controlPlaneMachineOptions() []client.ListOption {
+	return []client.ListOption{
+		client.HasLabels{clusterv1.MachineControlPlaneLabelName},
+	}
+}
diff --git a/test/framework/deployment_helpers.go b/test/framework/deployment_helpers.go
index 26cec38db..dc913077e 100644
--- a/test/framework/deployment_helpers.go
+++ b/test/framework/deployment_helpers.go
@@ -80,7 +80,7 @@ type WatchDeploymentLogsInput struct {
 func WatchDeploymentLogs(ctx context.Context, input WatchDeploymentLogsInput) {
 	Expect(ctx).NotTo(BeNil(), "ctx is required for WatchControllerLogs")
 	Expect(input.ClientSet).NotTo(BeNil(), "input.ClientSet is required for WatchControllerLogs")
-	Expect(input.Deployment).NotTo(BeNil(), "input.Name is required for WatchControllerLogs")
+	Expect(input.Deployment).NotTo(BeNil(), "input.Deployment is required for WatchControllerLogs")
 
 	deployment := &appsv1.Deployment{}
 	key, err := client.ObjectKeyFromObject(input.Deployment)
diff --git a/test/framework/machine_helpers.go b/test/framework/machine_helpers.go
index a49921f8c..1c8eb16e9 100644
--- a/test/framework/machine_helpers.go
+++ b/test/framework/machine_helpers.go
@@ -18,27 +18,89 @@ package framework
 
 import (
 	"context"
+	"fmt"
 
 	. "github.com/onsi/ginkgo"
 	. "github.com/onsi/gomega"
-	"github.com/pkg/errors"
 
+	"github.com/pkg/errors"
+	corev1 "k8s.io/api/core/v1"
+	"k8s.io/apimachinery/pkg/types"
 	clusterv1 "sigs.k8s.io/cluster-api/api/v1alpha3"
+	"sigs.k8s.io/cluster-api/util/patch"
 )
 
-// WaitForMachinesToBeUpgradedInput is the input for WaitForMachinesToBeUpgraded.
-type WaitForMachinesToBeUpgradedInput struct {
+// GetMachinesByMachineDeploymentsInput is the input for GetMachinesByMachineDeployments.
+type GetMachinesByMachineDeploymentsInput struct {
+	Lister            Lister
+	ClusterName       string
+	Namespace         string
+	MachineDeployment clusterv1.MachineDeployment
+}
+
+// GetMachinesByMachineDeployments returns Machine objects for a cluster belonging to a machine deployment.
+// Important! this method relies on labels that are created by the CAPI controllers during the first reconciliation, so
+// it is necessary to ensure this is already happened before calling it.
+func GetMachinesByMachineDeployments(ctx context.Context, input GetMachinesByMachineDeploymentsInput) []clusterv1.Machine {
+	Expect(ctx).NotTo(BeNil(), "ctx is required for GetMachinesByMachineDeployments")
+	Expect(input.Lister).ToNot(BeNil(), "Invalid argument. input.Lister can't be nil when calling GetMachinesByMachineDeployments")
+	Expect(input.ClusterName).ToNot(BeEmpty(), "Invalid argument. input.ClusterName can't be empty when calling GetMachinesByMachineDeployments")
+	Expect(input.Namespace).ToNot(BeEmpty(), "Invalid argument. input.Namespace can't be empty when calling GetMachinesByMachineDeployments")
+	Expect(input.MachineDeployment).ToNot(BeNil(), "Invalid argument. input.MachineDeployment can't be nil when calling GetMachinesByMachineDeployments")
+
+	opts := byClusterOptions(input.ClusterName, input.Namespace)
+	opts = append(opts, machineDeploymentOptions(input.MachineDeployment)...)
+
+	machineList := &clusterv1.MachineList{}
+	Expect(input.Lister.List(ctx, machineList, opts...)).To(Succeed(), "Failed to list MachineList object for Cluster %s/%s", input.Namespace, input.ClusterName)
+
+	return machineList.Items
+}
+
+// GetControlPlaneMachinesByClusterInput is the input for GetControlPlaneMachinesByCluster.
+type GetControlPlaneMachinesByClusterInput struct {
+	Lister      Lister
+	ClusterName string
+	Namespace   string
+}
+
+// GetControlPlaneMachinesByCluster returns the Machine objects for a cluster.
+// Important! this method relies on labels that are created by the CAPI controllers during the first reconciliation, so
+// it is necessary to ensure this is already happened before calling it.
+func GetControlPlaneMachinesByCluster(ctx context.Context, input GetControlPlaneMachinesByClusterInput) []clusterv1.Machine {
+	Expect(ctx).NotTo(BeNil(), "ctx is required for GetControlPlaneMachinesByCluster")
+	Expect(input.Lister).ToNot(BeNil(), "Invalid argument. input.Lister can't be nil when calling GetControlPlaneMachinesByCluster")
+	Expect(input.ClusterName).ToNot(BeEmpty(), "Invalid argument. input.ClusterName can't be empty when calling GetControlPlaneMachinesByCluster")
+	Expect(input.Namespace).ToNot(BeEmpty(), "Invalid argument. input.Namespace can't be empty when calling GetControlPlaneMachinesByCluster")
+
+	options := append(byClusterOptions(input.ClusterName, input.Namespace), controlPlaneMachineOptions()...)
+
+	machineList := &clusterv1.MachineList{}
+	Expect(input.Lister.List(ctx, machineList, options...)).To(Succeed(), "Failed to list MachineList object for Cluster %s/%s", input.Namespace, input.ClusterName)
+
+	return machineList.Items
+}
+
+// WaitForControlPlaneMachinesToBeUpgradedInput is the input for WaitForControlPlaneMachinesToBeUpgraded.
+type WaitForControlPlaneMachinesToBeUpgradedInput struct {
 	Lister                   Lister
 	Cluster                  *clusterv1.Cluster
 	KubernetesUpgradeVersion string
 	MachineCount             int
 }
 
-// WaitForMachinesToBeUpgraded waits until all machines are upgraded to the correct kubernetes version.
-func WaitForMachinesToBeUpgraded(ctx context.Context, input WaitForMachinesToBeUpgradedInput, intervals ...interface{}) {
+// WaitForControlPlaneMachinesToBeUpgraded waits until all machines are upgraded to the correct kubernetes version.
+func WaitForControlPlaneMachinesToBeUpgraded(ctx context.Context, input WaitForControlPlaneMachinesToBeUpgradedInput, intervals ...interface{}) {
+	Expect(ctx).NotTo(BeNil(), "ctx is required for WaitForControlPlaneMachinesToBeUpgraded")
+	Expect(input.Lister).ToNot(BeNil(), "Invalid argument. input.Lister can't be nil when calling WaitForControlPlaneMachinesToBeUpgraded")
+	Expect(input.KubernetesUpgradeVersion).ToNot(BeEmpty(), "Invalid argument. input.KubernetesUpgradeVersion can't be empty when calling WaitForControlPlaneMachinesToBeUpgraded")
+	Expect(input.MachineCount).To(BeNumerically(">", 0), "Invalid argument. input.MachineCount can't be smaller than 1 when calling WaitForControlPlaneMachinesToBeUpgraded")
+
 	By("ensuring all machines have upgraded kubernetes version")
+	fmt.Fprintf(GinkgoWriter, "Ensuring all MachineDeployment Machines have upgraded kubernetes version %s\n", input.KubernetesUpgradeVersion)
+
 	Eventually(func() (int, error) {
-		machines := GetControlPlaneMachinesByCluster(context.TODO(), GetMachinesByClusterInput{
+		machines := GetControlPlaneMachinesByCluster(context.TODO(), GetControlPlaneMachinesByClusterInput{
 			Lister:      input.Lister,
 			ClusterName: input.Cluster.Name,
 			Namespace:   input.Cluster.Namespace,
@@ -56,3 +118,69 @@ func WaitForMachinesToBeUpgraded(ctx context.Context, input WaitForMachinesToBeU
 		return upgraded, nil
 	}, intervals...).Should(Equal(input.MachineCount))
 }
+
+// WaitForMachineDeploymentMachinesToBeUpgradedInput is the input for WaitForMachineDeploymentMachinesToBeUpgraded.
+type WaitForMachineDeploymentMachinesToBeUpgradedInput struct {
+	Lister                   Lister
+	Cluster                  *clusterv1.Cluster
+	KubernetesUpgradeVersion string
+	MachineCount             int
+	MachineDeployment        clusterv1.MachineDeployment
+}
+
+// WaitForMachineDeploymentMachinesToBeUpgraded waits until all machines belonging to a MachineDeployment are upgraded to the correct kubernetes version.
+func WaitForMachineDeploymentMachinesToBeUpgraded(ctx context.Context, input WaitForMachineDeploymentMachinesToBeUpgradedInput, intervals ...interface{}) {
+	Expect(ctx).NotTo(BeNil(), "ctx is required for WaitForMachineDeploymentMachinesToBeUpgraded")
+	Expect(input.Lister).ToNot(BeNil(), "Invalid argument. input.Getter can't be nil when calling WaitForMachineDeploymentMachinesToBeUpgraded")
+	Expect(input.Cluster).ToNot(BeNil(), "Invalid argument. input.Cluster can't be nil when calling WaitForMachineDeploymentMachinesToBeUpgraded")
+	Expect(input.KubernetesUpgradeVersion).ToNot(BeNil(), "Invalid argument. input.KubernetesUpgradeVersion can't be nil when calling WaitForMachineDeploymentMachinesToBeUpgraded")
+	Expect(input.MachineDeployment).ToNot(BeNil(), "Invalid argument. input.MachineDeployment can't be nil when calling WaitForMachineDeploymentMachinesToBeUpgraded")
+	Expect(input.MachineCount).To(BeNumerically(">", 0), "Invalid argument. input.MachineCount can't be smaller than 1 when calling WaitForMachineDeploymentMachinesToBeUpgraded")
+
+	fmt.Fprintf(GinkgoWriter, "Ensuring all MachineDeployment Machines have upgraded kubernetes version %s\n", input.KubernetesUpgradeVersion)
+	Eventually(func() (int, error) {
+		machines := GetMachinesByMachineDeployments(context.TODO(), GetMachinesByMachineDeploymentsInput{
+			Lister:            input.Lister,
+			ClusterName:       input.Cluster.Name,
+			Namespace:         input.Cluster.Namespace,
+			MachineDeployment: input.MachineDeployment,
+		})
+
+		upgraded := 0
+		for _, machine := range machines {
+			if *machine.Spec.Version == input.KubernetesUpgradeVersion {
+				upgraded++
+			}
+		}
+		if len(machines) > upgraded {
+			return 0, errors.New("old nodes remain")
+		}
+		return upgraded, nil
+	}, intervals...).Should(Equal(input.MachineCount))
+}
+
+// PatchNodeConditionInput is the input for PatchNodeCondition.
+type PatchNodeConditionInput struct {
+	ClusterProxy  ClusterProxy
+	Cluster       *clusterv1.Cluster
+	NodeCondition corev1.NodeCondition
+	Machine       clusterv1.Machine
+}
+
+// PatchNodeCondition patches a node condition to any one of the machines with a node ref.
+func PatchNodeCondition(ctx context.Context, input PatchNodeConditionInput) {
+	Expect(ctx).NotTo(BeNil(), "ctx is required for PatchNodeConditions")
+	Expect(input.ClusterProxy).ToNot(BeNil(), "Invalid argument. input.ClusterProxy can't be nil when calling PatchNodeConditions")
+	Expect(input.Cluster).ToNot(BeNil(), "Invalid argument. input.Cluster can't be nil when calling PatchNodeConditions")
+	Expect(input.NodeCondition).ToNot(BeNil(), "Invalid argument. input.NodeCondition can't be nil when calling PatchNodeConditions")
+	Expect(input.Machine).ToNot(BeNil(), "Invalid argument. input.Machine can't be nil when calling PatchNodeConditions")
+
+	fmt.Fprintf(GinkgoWriter, "Patching the node condition to the node\n")
+	Expect(input.Machine.Status.NodeRef).ToNot(BeNil())
+	node := &corev1.Node{}
+	Expect(input.ClusterProxy.GetWorkloadCluster(ctx, input.Cluster.Namespace, input.Cluster.Name).GetClient().Get(ctx, types.NamespacedName{Name: input.Machine.Status.NodeRef.Name, Namespace: input.Machine.Status.NodeRef.Namespace}, node)).To(Succeed())
+	patchHelper, err := patch.NewHelper(node, input.ClusterProxy.GetWorkloadCluster(ctx, input.Cluster.Namespace, input.Cluster.Name).GetClient())
+	Expect(err).ToNot(HaveOccurred())
+	node.Status.Conditions = append(node.Status.Conditions, input.NodeCondition)
+	Expect(patchHelper.Patch(ctx, node)).To(Succeed())
+}
diff --git a/test/framework/machinedeployment_helpers.go b/test/framework/machinedeployment_helpers.go
index cd10dc1ab..d3f8f176f 100644
--- a/test/framework/machinedeployment_helpers.go
+++ b/test/framework/machinedeployment_helpers.go
@@ -18,14 +18,18 @@ package framework
 
 import (
 	"context"
+	"fmt"
 
 	. "github.com/onsi/ginkgo"
 	. "github.com/onsi/gomega"
-	"github.com/pkg/errors"
 
+	"github.com/pkg/errors"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"
 	"k8s.io/apimachinery/pkg/runtime"
 	clusterv1 "sigs.k8s.io/cluster-api/api/v1alpha3"
+	"sigs.k8s.io/cluster-api/util"
+	"sigs.k8s.io/cluster-api/util/patch"
 	"sigs.k8s.io/controller-runtime/pkg/client"
 )
 
@@ -79,6 +83,10 @@ type WaitForMachineDeploymentNodesToExistInput struct {
 
 // WaitForMachineDeploymentNodesToExist waits until all nodes associated with a machine deployment exist.
 func WaitForMachineDeploymentNodesToExist(ctx context.Context, input WaitForMachineDeploymentNodesToExistInput, intervals ...interface{}) {
+	Expect(ctx).NotTo(BeNil(), "ctx is required for WaitForMachineDeploymentNodesToExist")
+	Expect(input.Lister).ToNot(BeNil(), "Invalid argument. input.Lister can't be nil when calling WaitForMachineDeploymentNodesToExist")
+	Expect(input.MachineDeployment).ToNot(BeNil(), "Invalid argument. input.MachineDeployment can't be nil when calling WaitForMachineDeploymentNodesToExist")
+
 	By("waiting for the workload nodes to exist")
 	Eventually(func() (int, error) {
 		selectorMap, err := metav1.LabelSelectorAsMap(&input.MachineDeployment.Spec.Selector)
@@ -137,3 +145,147 @@ func DiscoveryAndWaitForMachineDeployments(ctx context.Context, input DiscoveryA
 	}
 	return machineDeployments
 }
+
+// UpgradeMachineDeploymentsAndWaitInput is the input type for UpgradeMachineDeploymentsAndWait.
+type UpgradeMachineDeploymentsAndWaitInput struct {
+	ClusterProxy                ClusterProxy
+	Cluster                     *clusterv1.Cluster
+	UpgradeVersion              string
+	MachineDeployments          []*clusterv1.MachineDeployment
+	WaitForMachinesToBeUpgraded []interface{}
+}
+
+// UpgradeMachineDeploymentsAndWait upgrades a machine deployment and waits for its machines to be upgraded.
+func UpgradeMachineDeploymentsAndWait(ctx context.Context, input UpgradeMachineDeploymentsAndWaitInput) {
+	Expect(ctx).NotTo(BeNil(), "ctx is required for UpgradeMachineDeploymentsAndWait")
+	Expect(input.ClusterProxy).ToNot(BeNil(), "Invalid argument. input.ClusterProxy can't be nil when calling UpgradeMachineDeploymentsAndWait")
+	Expect(input.Cluster).ToNot(BeNil(), "Invalid argument. input.Cluster can't be nil when calling UpgradeMachineDeploymentsAndWait")
+	Expect(input.UpgradeVersion).ToNot(BeNil(), "Invalid argument. input.UpgradeVersion can't be nil when calling UpgradeMachineDeploymentsAndWait")
+	Expect(input.MachineDeployments).ToNot(BeEmpty(), "Invalid argument. input.MachineDeployments can't be empty when calling UpgradeMachineDeploymentsAndWait")
+
+	mgmtClient := input.ClusterProxy.GetClient()
+
+	for _, deployment := range input.MachineDeployments {
+		fmt.Fprintf(GinkgoWriter, "Patching the new kubernetes version to Machine Deployment %s/%s\n", deployment.Namespace, deployment.Name)
+		patchHelper, err := patch.NewHelper(deployment, mgmtClient)
+		Expect(err).ToNot(HaveOccurred())
+
+		oldVersion := deployment.Spec.Template.Spec.Version
+		deployment.Spec.Template.Spec.Version = &input.UpgradeVersion
+		Expect(patchHelper.Patch(context.TODO(), deployment)).To(Succeed())
+
+		fmt.Fprintf(GinkgoWriter, "Waiting for Kubernetes versions of machines in MachineDeployment %s/%s to be upgraded from %s to %s\n",
+			deployment.Namespace, deployment.Name, *oldVersion, input.UpgradeVersion)
+		WaitForMachineDeploymentMachinesToBeUpgraded(ctx, WaitForMachineDeploymentMachinesToBeUpgradedInput{
+			Lister:                   mgmtClient,
+			Cluster:                  input.Cluster,
+			MachineCount:             int(*deployment.Spec.Replicas),
+			KubernetesUpgradeVersion: input.UpgradeVersion,
+			MachineDeployment:        *deployment,
+		}, input.WaitForMachinesToBeUpgraded...)
+	}
+}
+
+// WaitForMachineDeploymentRollingUpgradeToStartInput is the input for WaitForMachineDeploymentRollingUpgradeToStart.
+type WaitForMachineDeploymentRollingUpgradeToStartInput struct {
+	Getter            Getter
+	MachineDeployment *clusterv1.MachineDeployment
+}
+
+// WaitForMachineDeploymentRollingUpgradeToStart waits until rolling upgrade starts.
+func WaitForMachineDeploymentRollingUpgradeToStart(ctx context.Context, input WaitForMachineDeploymentRollingUpgradeToStartInput, intervals ...interface{}) {
+	Expect(ctx).NotTo(BeNil(), "ctx is required for WaitForMachineDeploymentRollingUpgradeToStart")
+	Expect(input.Getter).ToNot(BeNil(), "Invalid argument. input.Getter can't be nil when calling WaitForMachineDeploymentRollingUpgradeToStart")
+	Expect(input.MachineDeployment).ToNot(BeNil(), "Invalid argument. input.MachineDeployment can't be nil when calling WaitForMachineDeploymentRollingUpgradeToStarts")
+
+	fmt.Fprintf(GinkgoWriter, "Waiting for MachineDeployment rolling upgrade to start\n")
+	Eventually(func() bool {
+		md := &clusterv1.MachineDeployment{}
+		Expect(input.Getter.Get(ctx, client.ObjectKey{Namespace: input.MachineDeployment.Namespace, Name: input.MachineDeployment.Name}, md)).To(Succeed())
+		return md.Status.Replicas != md.Status.AvailableReplicas
+	}, intervals...).Should(BeTrue())
+}
+
+// WaitForMachineDeploymentRollingUpgradeToCompleteInput is the input for WaitForMachineDeploymentRollingUpgradeToComplete.
+type WaitForMachineDeploymentRollingUpgradeToCompleteInput struct {
+	Getter            Getter
+	MachineDeployment *clusterv1.MachineDeployment
+}
+
+// WaitForMachineDeploymentNodesToExist waits until rolling upgrade is complete.
+func WaitForMachineDeploymentRollingUpgradeToComplete(ctx context.Context, input WaitForMachineDeploymentRollingUpgradeToCompleteInput, intervals ...interface{}) {
+	Expect(ctx).NotTo(BeNil(), "ctx is required for WaitForMachineDeploymentRollingUpgradeToComplete")
+	Expect(input.Getter).ToNot(BeNil(), "Invalid argument. input.Getter can't be nil when calling WaitForMachineDeploymentRollingUpgradeToComplete")
+	Expect(input.MachineDeployment).ToNot(BeNil(), "Invalid argument. input.MachineDeployment can't be nil when calling WaitForMachineDeploymentRollingUpgradeToComplete")
+
+	fmt.Fprintf(GinkgoWriter, "Waiting for MachineDeployment rolling upgrade to complete\n")
+	Eventually(func() bool {
+		md := &clusterv1.MachineDeployment{}
+		Expect(input.Getter.Get(ctx, client.ObjectKey{Namespace: input.MachineDeployment.Namespace, Name: input.MachineDeployment.Name}, md)).To(Succeed())
+		return md.Status.Replicas == md.Status.AvailableReplicas
+	}, intervals...).Should(BeTrue())
+}
+
+// UpgradeMachineDeploymentInfrastructureRefAndWaitInput is the input type for UpgradeMachineDeploymentInfrastructureRefAndWait.
+type UpgradeMachineDeploymentInfrastructureRefAndWaitInput struct {
+	ClusterProxy                ClusterProxy
+	Cluster                     *clusterv1.Cluster
+	MachineDeployments          []*clusterv1.MachineDeployment
+	WaitForMachinesToBeUpgraded []interface{}
+}
+
+// UpgradeMachineDeploymentInfrastructureRefAndWait upgrades a machine deployment infrastructure ref and waits for its machines to be upgraded.
+func UpgradeMachineDeploymentInfrastructureRefAndWait(ctx context.Context, input UpgradeMachineDeploymentInfrastructureRefAndWaitInput) {
+	Expect(ctx).NotTo(BeNil(), "ctx is required for UpgradeMachineDeploymentInfrastructureRefAndWait")
+	Expect(input.ClusterProxy).ToNot(BeNil(), "Invalid argument. input.ClusterProxy can't be nil when calling UpgradeMachineDeploymentInfrastructureRefAndWait")
+	Expect(input.Cluster).ToNot(BeNil(), "Invalid argument. input.Cluster can't be nil when calling UpgradeMachineDeploymentInfrastructureRefAndWait")
+	Expect(input.MachineDeployments).ToNot(BeEmpty(), "Invalid argument. input.MachineDeployments can't be empty when calling UpgradeMachineDeploymentInfrastructureRefAndWait")
+
+	mgmtClient := input.ClusterProxy.GetClient()
+
+	for _, deployment := range input.MachineDeployments {
+		fmt.Fprintf(GinkgoWriter, "Patching the new infrastructure ref to Machine Deployment %s/%s\n", deployment.Namespace, deployment.Name)
+		// Retrieve infra object
+		infraRef := deployment.Spec.Template.Spec.InfrastructureRef
+		infraObj := &unstructured.Unstructured{}
+		infraObj.SetGroupVersionKind(infraRef.GroupVersionKind())
+		key := client.ObjectKey{
+			Namespace: input.Cluster.Namespace,
+			Name:      infraRef.Name,
+		}
+		Expect(mgmtClient.Get(ctx, key, infraObj)).NotTo(HaveOccurred())
+
+		// Creates a new infra object
+		newInfraObj := infraObj
+		newInfraObjName := fmt.Sprintf("%s-%s", infraRef.Name, util.RandomString(6))
+		newInfraObj.SetName(newInfraObjName)
+		newInfraObj.SetResourceVersion("")
+		Expect(mgmtClient.Create(ctx, newInfraObj)).NotTo(HaveOccurred())
+
+		// Patch the new infra object's ref to the machine deployment
+		patchHelper, err := patch.NewHelper(deployment, mgmtClient)
+		Expect(err).ToNot(HaveOccurred())
+		infraRef.Name = newInfraObjName
+		deployment.Spec.Template.Spec.InfrastructureRef = infraRef
+		Expect(patchHelper.Patch(context.TODO(), deployment)).To(Succeed())
+
+		fmt.Fprintf(GinkgoWriter, "Waiting for rolling upgrade to start.\n")
+		WaitForMachineDeploymentRollingUpgradeToStart(ctx, WaitForMachineDeploymentRollingUpgradeToStartInput{
+			Getter:            mgmtClient,
+			MachineDeployment: deployment,
+		}, input.WaitForMachinesToBeUpgraded...)
+
+		fmt.Fprintf(GinkgoWriter, "Waiting for rolling upgrade to complete.\n")
+		WaitForMachineDeploymentRollingUpgradeToComplete(ctx, WaitForMachineDeploymentRollingUpgradeToCompleteInput{
+			Getter:            mgmtClient,
+			MachineDeployment: deployment,
+		}, input.WaitForMachinesToBeUpgraded...)
+	}
+}
+
+// machineDeploymentOptions returns a set of ListOptions that allows to get all machine objects belonging to a machine deployment.
+func machineDeploymentOptions(deployment clusterv1.MachineDeployment) []client.ListOption {
+	return []client.ListOption{
+		client.MatchingLabels(deployment.Spec.Selector.MatchLabels),
+	}
+}
diff --git a/test/infrastructure/docker/controllers/dockercluster_controller.go b/test/infrastructure/docker/controllers/dockercluster_controller.go
index 4c0192aeb..2871a095a 100644
--- a/test/infrastructure/docker/controllers/dockercluster_controller.go
+++ b/test/infrastructure/docker/controllers/dockercluster_controller.go
@@ -27,7 +27,6 @@ import (
 	"sigs.k8s.io/cluster-api/test/infrastructure/docker/docker"
 	"sigs.k8s.io/cluster-api/util"
 	"sigs.k8s.io/cluster-api/util/patch"
-	"sigs.k8s.io/cluster-api/util/predicates"
 	ctrl "sigs.k8s.io/controller-runtime"
 	"sigs.k8s.io/controller-runtime/pkg/client"
 	"sigs.k8s.io/controller-runtime/pkg/controller/controllerutil"
@@ -151,18 +150,13 @@ func reconcileDelete(ctx context.Context, dockerCluster *infrav1.DockerCluster,
 
 // SetupWithManager will add watches for this controller
 func (r *DockerClusterReconciler) SetupWithManager(mgr ctrl.Manager) error {
-	c, err := ctrl.NewControllerManagedBy(mgr).
+	return ctrl.NewControllerManagedBy(mgr).
 		For(&infrav1.DockerCluster{}).
-		WithEventFilter(predicates.ResourceNotPaused(r.Log)).
-		Build(r)
-	if err != nil {
-		return err
-	}
-	return c.Watch(
-		&source.Kind{Type: &clusterv1.Cluster{}},
-		&handler.EnqueueRequestsFromMapFunc{
-			ToRequests: util.ClusterToInfrastructureMapFunc(infrav1.GroupVersion.WithKind("DockerCluster")),
-		},
-		predicates.ClusterUnpaused(r.Log),
-	)
+		Watches(
+			&source.Kind{Type: &clusterv1.Cluster{}},
+			&handler.EnqueueRequestsFromMapFunc{
+				ToRequests: util.ClusterToInfrastructureMapFunc(infrav1.GroupVersion.WithKind("DockerCluster")),
+			},
+		).
+		Complete(r)
 }
diff --git a/test/infrastructure/docker/controllers/dockermachine_controller.go b/test/infrastructure/docker/controllers/dockermachine_controller.go
index ad8d5c590..cae115234 100644
--- a/test/infrastructure/docker/controllers/dockermachine_controller.go
+++ b/test/infrastructure/docker/controllers/dockermachine_controller.go
@@ -31,7 +31,6 @@ import (
 	"sigs.k8s.io/cluster-api/test/infrastructure/docker/docker"
 	"sigs.k8s.io/cluster-api/util"
 	"sigs.k8s.io/cluster-api/util/patch"
-	"sigs.k8s.io/cluster-api/util/predicates"
 	ctrl "sigs.k8s.io/controller-runtime"
 	"sigs.k8s.io/controller-runtime/pkg/client"
 	"sigs.k8s.io/controller-runtime/pkg/controller"
@@ -252,15 +251,8 @@ func (r *DockerMachineReconciler) reconcileDelete(ctx context.Context, machine *
 
 // SetupWithManager will add watches for this controller
 func (r *DockerMachineReconciler) SetupWithManager(mgr ctrl.Manager, options controller.Options) error {
-	clusterToDockerMachines, err := util.ClusterToObjectsMapper(mgr.GetClient(), &infrav1.DockerMachineList{}, mgr.GetScheme())
-	if err != nil {
-		return err
-	}
-
-	c, err := ctrl.NewControllerManagedBy(mgr).
+	return ctrl.NewControllerManagedBy(mgr).
 		For(&infrav1.DockerMachine{}).
-		WithOptions(options).
-		WithEventFilter(predicates.ResourceNotPaused(r.Log)).
 		Watches(
 			&source.Kind{Type: &clusterv1.Machine{}},
 			&handler.EnqueueRequestsFromMapFunc{
@@ -273,17 +265,8 @@ func (r *DockerMachineReconciler) SetupWithManager(mgr ctrl.Manager, options con
 				ToRequests: handler.ToRequestsFunc(r.DockerClusterToDockerMachines),
 			},
 		).
-		Build(r)
-	if err != nil {
-		return err
-	}
-	return c.Watch(
-		&source.Kind{Type: &clusterv1.Cluster{}},
-		&handler.EnqueueRequestsFromMapFunc{
-			ToRequests: clusterToDockerMachines,
-		},
-		predicates.ClusterUnpausedAndInfrastructureReady(r.Log),
-	)
+		WithOptions(options).
+		Complete(r)
 }
 
 // DockerClusterToDockerMachines is a handler.ToRequestsFunc to be used to enqeue
diff --git a/test/infrastructure/docker/go.sum b/test/infrastructure/docker/go.sum
index 261f6c684..99a7cb227 100644
--- a/test/infrastructure/docker/go.sum
+++ b/test/infrastructure/docker/go.sum
@@ -161,6 +161,8 @@ github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfU
 github.com/golang/groupcache v0.0.0-20160516000752-02826c3e7903/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
 github.com/golang/groupcache v0.0.0-20190129154638-5b532d6fd5ef h1:veQD95Isof8w9/WXiA+pa3tz3fJXkt5B7QaRBrM62gk=
 github.com/golang/groupcache v0.0.0-20190129154638-5b532d6fd5ef/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
+github.com/golang/groupcache v0.0.0-20200121045136-8c9f03a8e57e h1:1r7pUrabqp18hOBcwBwiTsbnFeTZHV9eER/QT5JVZxY=
+github.com/golang/groupcache v0.0.0-20200121045136-8c9f03a8e57e/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
 github.com/golang/mock v1.1.1/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A=
 github.com/golang/mock v1.2.0/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A=
 github.com/golang/protobuf v0.0.0-20161109072736-4bd1920723d7/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
@@ -205,12 +207,16 @@ github.com/hashicorp/golang-lru v0.0.0-20180201235237-0fb14efe8c47/go.mod h1:/m3
 github.com/hashicorp/golang-lru v0.5.0/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8=
 github.com/hashicorp/golang-lru v0.5.1 h1:0hERBMJE1eitiLkihrMvRVBYAkpHzc/J3QdDN+dAcgU=
 github.com/hashicorp/golang-lru v0.5.1/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8=
+github.com/hashicorp/golang-lru v0.5.4 h1:YDjusn29QI/Das2iO9M0BHnIbxPeyuCHsjMW+lJfyTc=
+github.com/hashicorp/golang-lru v0.5.4/go.mod h1:iADmTwqILo4mZ8BN3D2Q6+9jd8WM5uGBxy+E8yxSoD4=
 github.com/hashicorp/hcl v1.0.0/go.mod h1:E5yfLk+7swimpb2L/Alb/PJmXilQ/rhwaUYs4T20WEQ=
 github.com/hpcloud/tail v1.0.0 h1:nfCOvKYfkgYP8hkirhJocXT2+zOD8yUNjXaWfTlyFKI=
 github.com/hpcloud/tail v1.0.0/go.mod h1:ab1qPbhIpdTxEkNHXyeSf5vhxWSCs/tWer42PpOxQnU=
 github.com/imdario/mergo v0.3.5/go.mod h1:2EnlNZ0deacrJVfApfmtdGgDfMuh/nq6Ok1EcJh5FfA=
 github.com/imdario/mergo v0.3.6 h1:xTNEAn+kxVO7dTZGu0CegyqKZmoWFI0rF8UxjlB2d28=
 github.com/imdario/mergo v0.3.6/go.mod h1:2EnlNZ0deacrJVfApfmtdGgDfMuh/nq6Ok1EcJh5FfA=
+github.com/imdario/mergo v0.3.8 h1:CGgOkSJeqMRmt0D9XLWExdT4m4F1vd3FV3VPt+0VxkQ=
+github.com/imdario/mergo v0.3.8/go.mod h1:2EnlNZ0deacrJVfApfmtdGgDfMuh/nq6Ok1EcJh5FfA=
 github.com/inconshreveable/mousetrap v1.0.0/go.mod h1:PxqpIevigyE2G7u3NXJIT2ANytuPF1OarO4DADm73n8=
 github.com/jimstudt/http-authentication v0.0.0-20140401203705-3eca13d6893a/go.mod h1:wK6yTYYcgjHE1Z1QtXACPDjcFJyBskHEdagmnq3vsP8=
 github.com/jonboulle/clockwork v0.1.0/go.mod h1:Ii8DK3G1RaLaWxj9trq07+26W01tbo22gdxWY5EU2bo=
@@ -397,8 +403,9 @@ golang.org/x/crypto v0.0.0-20190605123033-f99c8df09eb5/go.mod h1:yigFU9vqHzYiE8U
 golang.org/x/crypto v0.0.0-20190611184440-5c40567a22f8/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
 golang.org/x/crypto v0.0.0-20190617133340-57b3e21c3d56/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
 golang.org/x/crypto v0.0.0-20190820162420-60c769a6c586/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
-golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550 h1:ObdrDkeb4kJdCP557AjRjq69pTHfNouLtWZG7j9rPN8=
 golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
+golang.org/x/crypto v0.0.0-20200302210943-78000ba7a073 h1:xMPOj6Pz6UipU1wXLkrtqpHbR0AVFnyPEQq/wRWz9lM=
+golang.org/x/crypto v0.0.0-20200302210943-78000ba7a073/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
 golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
 golang.org/x/exp v0.0.0-20190125153040-c74c464bbbf2/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
 golang.org/x/exp v0.0.0-20190312203227-4b39c73a6495/go.mod h1:ZjyILWgesfNpC6sMxTJOJm9Kp84zZh5NQWvqDGG3Qr8=
@@ -474,6 +481,8 @@ golang.org/x/time v0.0.0-20180412165947-fbb02b2291d2/go.mod h1:tRJNPiyCQ0inRvYxb
 golang.org/x/time v0.0.0-20181108054448-85acf8d2951c/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
 golang.org/x/time v0.0.0-20190308202827-9d24e82272b4 h1:SvFZT6jyqRaOeXpc5h/JSfZenJ2O330aBsf7JfSUXmQ=
 golang.org/x/time v0.0.0-20190308202827-9d24e82272b4/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
+golang.org/x/time v0.0.0-20191024005414-555d28b269f0 h1:/5xXl8Y5W96D+TtHSlonuFqGHIWVuyCkGJLwGh9JJFs=
+golang.org/x/time v0.0.0-20191024005414-555d28b269f0/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
 golang.org/x/tools v0.0.0-20180221164845-07fd8470d635/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
 golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
 golang.org/x/tools v0.0.0-20181011042414-1f849cf54d09/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=