Repository: syself/cluster-api-provider-hetzner
Pull request #1459 (2024-08-21 12:19:04)

:seedling: fix flaky unit-tests
Avoid global hcloudClient. The hcloudClient got created once, and stored in a global variable. If the previous test left a fake hcloud-server in the client, then the next test sees a different result from hcloudClient.ListServers().


guettli: > extended explanation: this unfortunately won't help. What would help is initializing one factory per test, but that is almost impossible, since you need that factory shared across all controllers, i.e. your whole envtest environment.

Thank you for your hint. 

Since I avoid global variables as much as I can, I was not thinking that the global hcloudClient used a second global variable.

Afaik there is no way to avoid a global variable, if you want to initialize something in BeforeSuite.

I changed the PR. The former Close() method is not called Reset() and called before the tests get executed.

Example:

```go
			BeforeEach(func() {
				hcloudClient = testEnv.ResetAndGetHCloudClient()
```

This way we are sure that every test resets the cache.

Is that ok for you?
guettli: > This will work. However, I don't really like the naming. "Reset" is, as you write, not a general usable method, but only related to the fact that we clear a cache that we have in the fake client.
> 
> Therefore, I would prefer to have the "close" function, which is a general pattern, and use it in the "aftereach" function instead of the "beforeeach".

> If you don't want to change that, then okay.

I have reasons for that :-)

I have seen these conversations in previous jobs. I think it is better if a test ensures during beforeEach (in other frameworks called SetUp) that everything is the way the tests want it to be.
A different test could fail badly, so that the test was not able to clean up after itself. Then maybe 20 tests will fail because the precondition does not fit. Then it is hard to detect what was wrong. That's why I prefer to ensure a clean test environment in beforeEach.

> Are you sure though that we cannot use the global hcloudclient? I think you always get a seemingly new one in the individual tests, even though they aren't actually new, because they share a same global variable. Therefore, I think it would be less misleading if we actually do the same and keep the hcloudclient a global variable in the controllers/ tests. If I understand it correctly, it shouldn't change anything.

I like to avoid a global client, because the explit way makes it is less likely that a developer forgets to call Reset().

> Please just respond to the above points and then we can get this merged. It is anyway a good conclusion that you had with investigating the flaky tests!

@janiskemper Was I able to change your mind?

janiskemper: I think you didn't get my point :D Because I agree with you in general, however:
- we have a global client whether you want it or not. By defining new clients constantly you just always overwrite the same global client. You just pretend that you define new clients. That's what IMO confuses people and makes them believe (just as you did some days ago) that there is no global client and we can just redefine it however we want. I don't want to pretend that there is no global client.
- I also in theory agree with this BeforeEach is better than AfterEach. However, also doesn't matter here, as it is a global client that is shared among all. In theory, which is not the case for us, if you run these in parallel, they would reset the global client during the test of another one. But there I have no strong feelings. Here I am not happy with the naming, which is better when we do it at the end. But that is okay. The above point is the more relevant one
guettli: > I think you didn't get my point :D Because I agree with you in general, however:
> 

> * we have a global client whether you want it or not. By defining new clients constantly you just always overwrite the same global client. You just pretend that you define new clients. That's what IMO confuses people and makes them believe (just as you did some days ago) that there is no global client and we can just redefine it however we want. I don't want to pretend that there is no global client.

This variable is no longer global:

```
hcloudClient hcloudclient.Client
```

And the name of the method (`testEnv.ResetAndGetHCloudClient()`) make it very clear that something gets re-used.

@janiskemper I renamed the method to ResetAndGetGlobalHCloudClient. This makes it more explicit. What do you think?

> * I also in theory agree with this BeforeEach is better than AfterEach. However, also doesn't matter here, as it is a global client that is shared among all. In theory, which is not the case for us, if you run these in parallel, they would reset the global client during the test of another one. But there I have no strong feelings. Here I am not happy with the naming, which is better when we do it at the end. But that is okay. The above point is the more relevant one

> ...if you run these in parallel,...

My changes are not related to that. This is a different topic. If we want parallel execution, we need to call the tests via the ginkgo binary. It uses different processes, so that a global variable would be ok.

Patch:
diff --git a/.github/actions/manager-image/action.yaml b/.github/actions/manager-image/action.yaml
index 838ffc9c..65f51475 100644
--- a/.github/actions/manager-image/action.yaml
+++ b/.github/actions/manager-image/action.yaml
@@ -44,7 +44,7 @@ runs:
 
     # Import GitHub's cache build to docker cache
     - name: Copy Caph Golang cache to docker cache
-      uses: docker/build-push-action@16ebe778df0e7752d2cfcbd924afdbbd89c1a755 # v6.6.1
+      uses: docker/build-push-action@5cd11c3a4ced054e52742c5fd54dca954e0edd85 # v6.7.0
       with:
         provenance: false
         context: /tmp/.cache/caph
@@ -54,7 +54,7 @@ runs:
         target: import-cache
 
     - name: Build and push manager image
-      uses: docker/build-push-action@16ebe778df0e7752d2cfcbd924afdbbd89c1a755 # v6
+      uses: docker/build-push-action@5cd11c3a4ced054e52742c5fd54dca954e0edd85 # v6
       with:
         provenance: false
         context: .
diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml
index 745ec36f..661b93d6 100644
--- a/.github/workflows/build.yml
+++ b/.github/workflows/build.yml
@@ -88,7 +88,7 @@ jobs:
 
       # Import GitHub's cache build to docker cache
       - name: Copy Caph Golang cache to docker cache
-        uses: docker/build-push-action@16ebe778df0e7752d2cfcbd924afdbbd89c1a755 # v6.6.1
+        uses: docker/build-push-action@5cd11c3a4ced054e52742c5fd54dca954e0edd85 # v6.7.0
         with:
           provenance: false
           context: /tmp/.cache/caph
@@ -98,7 +98,7 @@ jobs:
           target: import-cache
 
       - name: Build and push manager image
-        uses: docker/build-push-action@16ebe778df0e7752d2cfcbd924afdbbd89c1a755 # v6
+        uses: docker/build-push-action@5cd11c3a4ced054e52742c5fd54dca954e0edd85 # v6
         id: docker_build_release
         with:
           provenance: false
@@ -150,7 +150,7 @@ jobs:
       # Store docker's golang's cache build locally only on the main branch
       - name: Store Caph Golang cache build locally
         if: ${{ steps.cache.outputs.cache-hit != 'true' }}
-        uses: docker/build-push-action@16ebe778df0e7752d2cfcbd924afdbbd89c1a755 # v6.6.1
+        uses: docker/build-push-action@5cd11c3a4ced054e52742c5fd54dca954e0edd85 # v6.7.0
         with:
           provenance: false
           context: .
diff --git a/.github/workflows/pr-verify.yml b/.github/workflows/pr-verify.yml
index f5be1421..a9f6bfdd 100644
--- a/.github/workflows/pr-verify.yml
+++ b/.github/workflows/pr-verify.yml
@@ -48,7 +48,7 @@ jobs:
           app-id: ${{ secrets.SYSELF_APP_ID }}
           private-key: ${{ secrets.SYSELF_APP_PRIVATE_KEY }}
       - name: Generate Size
-        uses: pascalgn/size-label-action@bbbaa0d5ccce8e2e76254560df5c64b82dac2e12 # v0.5.2
+        uses: pascalgn/size-label-action@be08a2d5f857dc99c5b9426cdb5a8ea1aa7f0399 # v0.5.4
         env:
           GITHUB_TOKEN: ${{ steps.generate-token.outputs.token }}
         with:
diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml
index 7cc1d9d7..6c4a51fa 100644
--- a/.github/workflows/release.yml
+++ b/.github/workflows/release.yml
@@ -60,7 +60,7 @@ jobs:
           echo 'EOF' >> $GITHUB_ENV
 
       - name: Build and push manager image
-        uses: docker/build-push-action@16ebe778df0e7752d2cfcbd924afdbbd89c1a755 # v6
+        uses: docker/build-push-action@5cd11c3a4ced054e52742c5fd54dca954e0edd85 # v6
         id: docker_build_release
         with:
           provenance: false
diff --git a/.github/workflows/schedule-update-bot.yaml b/.github/workflows/schedule-update-bot.yaml
index 52f7fccb..37165b84 100644
--- a/.github/workflows/schedule-update-bot.yaml
+++ b/.github/workflows/schedule-update-bot.yaml
@@ -44,7 +44,7 @@ jobs:
           echo "LOG_LEVEL=${{ github.event.inputs.logLevel || env.LOG_LEVEL }}" >> "$GITHUB_ENV"
 
       - name: Renovate
-        uses: renovatebot/github-action@0c94129d48b4e92addf23686b8a735b3057a0448 # v40.2.5
+        uses: renovatebot/github-action@630a255a1f2f56c8d8ce160bed3e3ca577ca53e2 # v40.2.7
         env:
           RENOVATE_HOST_RULES: '[{"hostType": "docker", "matchHost": "ghcr.io", "username": "${{ github.actor }}", "password": "${{ secrets.GITHUB_TOKEN }}" }]'
           RENOVATE_ALLOWED_POST_UPGRADE_COMMANDS: '[".*"]'
diff --git a/api/v1beta1/hcloudremediation_types.go b/api/v1beta1/hcloudremediation_types.go
index 07d1920b..695b1451 100644
--- a/api/v1beta1/hcloudremediation_types.go
+++ b/api/v1beta1/hcloudremediation_types.go
@@ -21,11 +21,6 @@ import (
 	clusterv1 "sigs.k8s.io/cluster-api/api/v1beta1"
 )
 
-const (
-	// HCloudRebootAnnotation indicates that a bare metal host object should be rebooted.
-	HCloudRebootAnnotation = "reboot.hcloud.infrastructure.cluster.x-k8s.io"
-)
-
 // HCloudRemediationSpec defines the desired state of HCloudRemediation.
 type HCloudRemediationSpec struct {
 	// Strategy field defines remediation strategy.
diff --git a/api/v1beta1/hetznerbaremetalhost_types.go b/api/v1beta1/hetznerbaremetalhost_types.go
index 801ded91..496f3fff 100644
--- a/api/v1beta1/hetznerbaremetalhost_types.go
+++ b/api/v1beta1/hetznerbaremetalhost_types.go
@@ -21,7 +21,6 @@ import (
 	"crypto/sha256"
 	"encoding/json"
 	"fmt"
-	"strings"
 	"time"
 
 	corev1 "k8s.io/api/core/v1"
@@ -46,10 +45,10 @@ const (
 
 	// WipeDiskAnnotation indicates which Disks (WWNs) to erase before provisioning
 	// The value is a list of WWNS or "all".
-	WipeDiskAnnotation = "wipedisk.hetznerbaremetalhost.infrastructure.cluster.x-k8s.io"
+	WipeDiskAnnotation = "capi.syself.com/wipe-disk"
 
 	// IgnoreCheckDiskAnnotation indicates that the machine should get provisioned, even if CheckDisk fails.
-	IgnoreCheckDiskAnnotation = "ignore-cd.hetznerbaremetalhost.infrastructure.cluster.x-k8s.io" // max length is 63 chars.
+	IgnoreCheckDiskAnnotation = "capi.syself.com/ignore-check-disk"
 )
 
 // RootDeviceHints holds the hints for specifying the storage location
@@ -602,7 +601,7 @@ func (host *HetznerBareMetalHost) ClearRebootAnnotations() {
 }
 
 func isRebootAnnotation(annotation string) bool {
-	return strings.HasPrefix(annotation, RebootAnnotation+"/") || annotation == RebootAnnotation
+	return annotation == RebootAnnotation
 }
 
 //+kubebuilder:object:root=true
diff --git a/api/v1beta1/hetznerbaremetalremediation_types.go b/api/v1beta1/hetznerbaremetalremediation_types.go
index 8fb7967f..4dd47d88 100644
--- a/api/v1beta1/hetznerbaremetalremediation_types.go
+++ b/api/v1beta1/hetznerbaremetalremediation_types.go
@@ -22,12 +22,12 @@ import (
 
 const (
 	// RebootAnnotation indicates that a bare metal host object should be rebooted.
-	RebootAnnotation = "reboot.hetznerbaremetalhost.infrastructure.cluster.x-k8s.io"
+	RebootAnnotation = "capi.syself.com/reboot"
 
 	// PermanentErrorAnnotation indicates that the bare metal host has an error which needs to be resolved manually.
 	// After the permanent error the annotation got removed (usually by a human), the controller removes
 	// ErrorType, ErrorCount and ErrorMessages, so that the hbmh will be usable again.
-	PermanentErrorAnnotation = "error.hetznerbaremetalhost.infrastructure.cluster.x-k8s.io"
+	PermanentErrorAnnotation = "capi.syself.com/permanent-error"
 )
 
 // HetznerBareMetalRemediationSpec defines the desired state of HetznerBareMetalRemediation.
diff --git a/api/v1beta1/hetznercluster_webhook.go b/api/v1beta1/hetznercluster_webhook.go
index 3904e459..72c0746e 100644
--- a/api/v1beta1/hetznercluster_webhook.go
+++ b/api/v1beta1/hetznercluster_webhook.go
@@ -74,10 +74,7 @@ func (r *HetznerCluster) ValidateCreate() (admission.Warnings, error) {
 	hetznerclusterlog.V(1).Info("validate create", "name", r.Name)
 	var allErrs field.ErrorList
 
-	allowEmptyControlPlaneAddress := false
-	if r.Annotations != nil {
-		allowEmptyControlPlaneAddress = r.Annotations[AllowEmptyControlPlaneAddressAnnotation] == "true"
-	}
+	allowEmptyControlPlaneAddress := r.Annotations[AllowEmptyControlPlaneAddressAnnotation] == "true"
 
 	if !allowEmptyControlPlaneAddress && len(r.Spec.ControlPlaneRegions) == 0 {
 		allErrs = append(allErrs, field.Invalid(
diff --git a/docs/README.md b/docs/README.md
index a76ea2fb..087cb30e 100644
--- a/docs/README.md
+++ b/docs/README.md
@@ -26,6 +26,7 @@ This is the official documentation of Cluster API Provider Hetzner. Before start
 - [HetznerBareMetalHost](/docs/caph/03-reference/05-hetzner-bare-metal-host.md)
 - [HetznerBareMetalMachineTemplate](/docs/caph/03-reference/06-hetzner-bare-metal-machine-template.md)
 - [HetznerBareMetalRemediationTemplate](/docs/caph/03-reference/07-hetzner-bare-metal-remediation-template.md)
+- [Annotations](/docs/caph/03-reference/08-annotations.md)
 
 ## Development
 
diff --git a/docs/caph/03-reference/08-annotations.md b/docs/caph/03-reference/08-annotations.md
new file mode 100644
index 00000000..f9dd6c3a
--- /dev/null
+++ b/docs/caph/03-reference/08-annotations.md
@@ -0,0 +1,55 @@
+---
+
+title: Annotations
+
+---
+
+You can set [Kubernetes Annotations](https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/) to instruct the Syself CAPH Controller to modify its behavior.
+
+## Overview of Annotations
+
+### capi.syself.com/wipe-disk
+
+| **Resource**    | [HetznerBareMetalHost](/docs/caph/03-reference/05-hetzner-bare-metal-host.md)                                                                                                                                                                                                                                       |
+| --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| **Description** | This annotation instructs the Syself CAPH Controller to wipe the disk before provisioning the machine.                                                                                                                                                                                                              |
+| **Value**       | You can use the string `"all"` or a space-separated list of WWNs. For example, `"10:00:00:05:1e:7a:7a:00 eui.00253885910c8cec 0x500a07511bb48b25"`. If the value is empty, no disks will be wiped. The value `"all"` will wipe all disks on the bare-metal machine (not just the one given in the rootDeviceHints). |
+| **Auto-Remove** | Enabled: The annotation is removed after the disks are wiped.                                                                                                                                                                                                                                                       |
+
+### capi.syself.com/ignore-check-disk
+
+| **Resource**    | [HetznerBareMetalHost](/docs/caph/03-reference/05-hetzner-bare-metal-host.md)                                                                                                                                                                                                                         |
+| --------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| **Description** | This annotation instructs the Syself CAPH Controller to ignore the results of the check-disk step during machine provisioning. The check will be performed, and a Kubernetes Event will be created. However, if this annotation is set, provisioning will continue even if a faulty disk is detected. |
+| **Value**       | The value is ignored. If the annotation exists, this feature is enabled.                                                                                                                                                                                                                              |
+| **Auto-Remove** | Disabled: The annotation remains on the resource. It is up to the user to remove it.                                                                                                                                                                                                                  |
+
+### capi.syself.com/allow-empty-control-plane-address
+
+| **Resource**    | [HetznerCluster](/docs/caph/03-reference/02-hetzner-cluster.md)                                                            |
+| --------------- | -------------------------------------------------------------------------------------------------------------------------- |
+| **Description** | This annotation allows the Syself CAPH Controller to create HetznerCluster resources with an empty `controlPlaneEndpoint`. This is useful to support externally managed control planes ([PR 1106](https://github.com/syself/cluster-api-provider-hetzner/pull/1106)) |
+| **Value**       | `"true"` enables this feature. All other strings are considered `"false"`.                                                 |
+| **Auto-Remove** | Disabled: The annotation remains on the resource.                                                                          |
+
+### capi.syself.com/constant-bare-metal-hostname
+
+| **Resource**    | Cluster (CRD of Cluster-API), HetznerBareMetalMachine                                                        |
+| --------------- | ------------------------------------------------------------------------------------------------------------ |
+| **Description** | See [Using constant hostnames](/docs/caph/02-topics/05-baremetal/04-constant-hostnames.md) for more details. |
+| **Auto-Remove** | Disabled: The annotation remains on the resource.                                                            |
+
+### capi.syself.com/reboot
+
+| **Resource**    | [HetznerBareMetalHost](/docs/caph/03-reference/05-hetzner-bare-metal-host.md)                                                                                                                                                                                 |
+| --------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| **Description** | If this annotation is present, the bare-metal machine will be rebooted. This annotation is used by `HetznerBareMetalRemediation` (see [Machine Health Checks with Custom Remediation Template](/docs/caph/02-topics/06-advanced/04-custom-templates-mhc.md)). |
+| **Value**       | The value is ignored. If the annotation exists, this feature is enabled.                                                                                                                                                                                      |
+| **Auto-Remove** | Enabled: The annotation is removed after the reboot.                                                                                                                                                                                                          |
+
+### capi.syself.com/permanent-error
+
+| **Resource**    | [HetznerBareMetalHost](/docs/caph/03-reference/05-hetzner-bare-metal-host.md)                                                                                                                                                                                                                                                                |
+| --------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| **Description** | This annotation is set by the Syself CAPH Controller when a bare-metal machine enters the "permanent error" state. This indicates that human intervention is required (e.g., to fix a broken disk). After the root cause is resolved, the user must remove this annotation to allow the Controller to manage the HetznerBareMetalHost again. |
+| **Auto-Remove** | Disabled: The annotation must be removed by the user.                                                                                                                                                                                                                                                                                        |
diff --git a/images/caph/Dockerfile b/images/caph/Dockerfile
index 25783f2f..ffbbc614 100644
--- a/images/caph/Dockerfile
+++ b/images/caph/Dockerfile
@@ -13,7 +13,7 @@
 # limitations under the License.
 
 # Build the manager binary
-FROM --platform=${BUILDPLATFORM} docker.io/library/golang:1.22.6-bullseye@sha256:825f81571780b54e553c4d7f21081efd47cd125dd7a8b3343a4591aa02f4816a AS build
+FROM --platform=${BUILDPLATFORM} docker.io/library/golang:1.22.6-bullseye@sha256:0838740c99e64dde1916553b004bdb1bc321823be83bde3d2a1563882eaab7e6 AS build
 ARG TARGETOS TARGETARCH
 
 COPY . /src/cluster-api-provider-hetzner
diff --git a/pkg/services/hcloud/client/client.go b/pkg/services/hcloud/client/client.go
index fa04d485..10c1fb12 100644
--- a/pkg/services/hcloud/client/client.go
+++ b/pkg/services/hcloud/client/client.go
@@ -40,7 +40,8 @@ var ErrUnauthorized = fmt.Errorf("unauthorized")
 
 // Client collects all methods used by the controller in the hcloud cloud API.
 type Client interface {
-	Close()
+	// Reset resets the local cache. Only implemented in the fake client.
+	Reset() Client
 
 	CreateLoadBalancer(context.Context, hcloud.LoadBalancerCreateOpts) (*hcloud.LoadBalancer, error)
 	DeleteLoadBalancer(context.Context, int64) error
@@ -78,6 +79,7 @@ type Client interface {
 
 // Factory is the interface for creating new Client objects.
 type Factory interface {
+	// NewClient returns a new Client in the real implementation, and the shared global Client in the fake implementation.
 	NewClient(hcloudToken string) Client
 }
 
@@ -141,8 +143,10 @@ type realClient struct {
 	client *hcloud.Client
 }
 
-// Close implements the Close method of the HCloudClient interface.
-func (c *realClient) Close() {}
+// Reset implements the Reset method of the HCloudClient interface.
+func (c *realClient) Reset() Client {
+	return c
+}
 
 func (c *realClient) CreateLoadBalancer(ctx context.Context, opts hcloud.LoadBalancerCreateOpts) (*hcloud.LoadBalancer, error) {
 	res, _, err := c.client.LoadBalancer.Create(ctx, opts)
diff --git a/pkg/services/hcloud/client/fake/hcloud_client.go b/pkg/services/hcloud/client/fake/hcloud_client.go
index 54229b17..62eddf8a 100644
--- a/pkg/services/hcloud/client/fake/hcloud_client.go
+++ b/pkg/services/hcloud/client/fake/hcloud_client.go
@@ -55,8 +55,8 @@ func (f *cacheHCloudClientFactory) NewClient(string) hcloudclient.Client {
 	return cacheHCloudClientInstance
 }
 
-// Close implements Close method of hcloud client interface.
-func (c *cacheHCloudClient) Close() {
+// Reset implements Reset method of hcloud client interface.
+func (c *cacheHCloudClient) Reset() hcloudclient.Client {
 	c.counterMutex.Lock()
 	defer c.counterMutex.Unlock()
 
@@ -86,6 +86,7 @@ func (c *cacheHCloudClient) Close() {
 	cacheHCloudClientInstance.placementGroupIDCounter = 0
 	cacheHCloudClientInstance.loadBalancerIDCounter = 0
 	cacheHCloudClientInstance.networkIDCounter = 0
+	return c
 }
 
 type cacheHCloudClientFactory struct{}
diff --git a/pkg/services/hcloud/client/mocks/Client.go b/pkg/services/hcloud/client/mocks/Client.go
index fd561434..5b321eb4 100644
--- a/pkg/services/hcloud/client/mocks/Client.go
+++ b/pkg/services/hcloud/client/mocks/Client.go
@@ -6,6 +6,7 @@ import (
 	context "context"
 
 	hcloud "github.com/hetznercloud/hcloud-go/v2/hcloud"
+	hcloudclient "github.com/syself/cluster-api-provider-hetzner/pkg/services/hcloud/client"
 
 	mock "github.com/stretchr/testify/mock"
 
@@ -161,11 +162,6 @@ func (_m *Client) ChangeLoadBalancerType(_a0 context.Context, _a1 *hcloud.LoadBa
 	return r0
 }
 
-// Close provides a mock function with given fields:
-func (_m *Client) Close() {
-	_m.Called()
-}
-
 // CreateLoadBalancer provides a mock function with given fields: _a0, _a1
 func (_m *Client) CreateLoadBalancer(_a0 context.Context, _a1 hcloud.LoadBalancerCreateOpts) (*hcloud.LoadBalancer, error) {
 	ret := _m.Called(_a0, _a1)
@@ -718,6 +714,26 @@ func (_m *Client) RebootServer(_a0 context.Context, _a1 *hcloud.Server) error {
 	return r0
 }
 
+// Reset provides a mock function with given fields:
+func (_m *Client) Reset() hcloudclient.Client {
+	ret := _m.Called()
+
+	if len(ret) == 0 {
+		panic("no return value specified for Reset")
+	}
+
+	var r0 hcloudclient.Client
+	if rf, ok := ret.Get(0).(func() hcloudclient.Client); ok {
+		r0 = rf()
+	} else {
+		if ret.Get(0) != nil {
+			r0 = ret.Get(0).(hcloudclient.Client)
+		}
+	}
+
+	return r0
+}
+
 // ShutdownServer provides a mock function with given fields: _a0, _a1
 func (_m *Client) ShutdownServer(_a0 context.Context, _a1 *hcloud.Server) error {
 	ret := _m.Called(_a0, _a1)
diff --git a/pkg/services/hcloud/server/server.go b/pkg/services/hcloud/server/server.go
index bce934e0..7911f708 100644
--- a/pkg/services/hcloud/server/server.go
+++ b/pkg/services/hcloud/server/server.go
@@ -189,6 +189,23 @@ func (s *Service) Reconcile(ctx context.Context) (res reconcile.Result, err erro
 	return res, nil
 }
 
+// implements setting rate limit on hcloudmachine.
+func handleRateLimit(hm *infrav1.HCloudMachine, err error, functionName string, errMsg string) error {
+	// returns error if not a rate limit exceeded error
+	if !hcloud.IsError(err, hcloud.ErrorCodeRateLimitExceeded) {
+		return fmt.Errorf("%s: %w", errMsg, err)
+	}
+
+	// does not return error if machine is running and does not have a deletion timestamp
+	if hm.Status.Ready && hm.DeletionTimestamp.IsZero() {
+		return nil
+	}
+
+	// check for a rate limit exceeded error if the machine is not running or if machine has a deletion timestamp
+	hcloudutil.HandleRateLimitExceeded(hm, err, functionName)
+	return fmt.Errorf("%s: %w", errMsg, err)
+}
+
 // Delete implements delete method of server.
 func (s *Service) Delete(ctx context.Context) (res reconcile.Result, err error) {
 	server, err := s.findServer(ctx)
@@ -246,12 +263,11 @@ func (s *Service) reconcileNetworkAttachment(ctx context.Context, server *hcloud
 			ID: s.scope.HetznerCluster.Status.Network.ID,
 		},
 	}); err != nil {
-		hcloudutil.HandleRateLimitExceeded(s.scope.HCloudMachine, err, "AttachServerToNetwork")
 		// check if network status is old and server is in fact already attached
 		if hcloud.IsError(err, hcloud.ErrorCodeServerAlreadyAttached) {
 			return nil
 		}
-		return fmt.Errorf("failed to attach server to network: %w", err)
+		return handleRateLimit(s.scope.HCloudMachine, err, "AttachServerToNetwork", "failed to attach server to network")
 	}
 
 	return nil
@@ -308,11 +324,11 @@ func (s *Service) reconcileLoadBalancerAttachment(ctx context.Context, server *h
 	}
 
 	if err := s.scope.HCloudClient.AddTargetServerToLoadBalancer(ctx, opts, loadBalancer); err != nil {
-		hcloudutil.HandleRateLimitExceeded(s.scope.HCloudMachine, err, "AddTargetServerToLoadBalancer")
 		if hcloud.IsError(err, hcloud.ErrorCodeTargetAlreadyDefined) {
 			return reconcile.Result{}, nil
 		}
-		return reconcile.Result{}, fmt.Errorf("failed to add server %s with ID %d as target to load balancer: %w", server.Name, server.ID, err)
+		errMsg := fmt.Sprintf("failed to add server %s with ID %d as target to load balancer", server.Name, server.ID)
+		return reconcile.Result{}, handleRateLimit(s.scope.HCloudMachine, err, "AddTargetServerToLoadBalancer", errMsg)
 	}
 
 	record.Eventf(
@@ -429,8 +445,7 @@ func (s *Service) createServer(ctx context.Context) (*hcloud.Server, error) {
 	// get all ssh keys that are stored in HCloud API
 	sshKeysAPI, err := s.scope.HCloudClient.ListSSHKeys(ctx, hcloud.SSHKeyListOpts{})
 	if err != nil {
-		hcloudutil.HandleRateLimitExceeded(s.scope.HCloudMachine, err, "ListSSHKeys")
-		return nil, fmt.Errorf("failed listing ssh heys from hcloud: %w", err)
+		return nil, handleRateLimit(s.scope.HCloudMachine, err, "ListSSHKeys", "failed listing ssh keys from hcloud")
 	}
 
 	// find matching keys and store them
@@ -479,8 +494,8 @@ func (s *Service) createServer(ctx context.Context) (*hcloud.Server, error) {
 			s.scope.Name(),
 			err,
 		)
-		err = errors.Join(errServerCreateNotPossible, err)
-		return nil, fmt.Errorf("failed to create HCloud server %s: %w", s.scope.HCloudMachine.Name, err)
+		errMsg := fmt.Sprintf("failed to create HCloud server %s", s.scope.HCloudMachine.Name)
+		return nil, handleRateLimit(s.scope.HCloudMachine, err, "CreateServer", errMsg)
 	}
 
 	// set ssh keys to status
@@ -497,8 +512,7 @@ func (s *Service) getServerImage(ctx context.Context) (*hcloud.Image, error) {
 	// Get server type so we can filter for images with correct architecture
 	serverType, err := s.scope.HCloudClient.GetServerType(ctx, string(s.scope.HCloudMachine.Spec.Type))
 	if err != nil {
-		hcloudutil.HandleRateLimitExceeded(s.scope.HCloudMachine, err, "GetServerType")
-		return nil, fmt.Errorf("failed to get server type in HCloud: %w", err)
+		return nil, handleRateLimit(s.scope.HCloudMachine, err, "GetServerType", "failed to get server type in HCloud")
 	}
 	if serverType == nil {
 		conditions.MarkFalse(
@@ -522,8 +536,7 @@ func (s *Service) getServerImage(ctx context.Context) (*hcloud.Image, error) {
 
 	images, err := s.scope.HCloudClient.ListImages(ctx, listOpts)
 	if err != nil {
-		hcloudutil.HandleRateLimitExceeded(s.scope.HCloudMachine, err, "ListImages")
-		return nil, fmt.Errorf("failed to list images by label in HCloud: %w", err)
+		return nil, handleRateLimit(s.scope.HCloudMachine, err, "ListImages", "failed to list images by label in HCloud")
 	}
 
 	// query for an existing image by name.
@@ -533,8 +546,7 @@ func (s *Service) getServerImage(ctx context.Context) (*hcloud.Image, error) {
 	}
 	imagesByName, err := s.scope.HCloudClient.ListImages(ctx, listOpts)
 	if err != nil {
-		hcloudutil.HandleRateLimitExceeded(s.scope.HCloudMachine, err, "ListImages")
-		return nil, fmt.Errorf("failed to list images by name in HCloud: %w", err)
+		return nil, handleRateLimit(s.scope.HCloudMachine, err, "ListImages", "failed to list images by name in HCloud")
 	}
 
 	images = append(images, imagesByName...)
@@ -577,12 +589,11 @@ func (s *Service) handleServerStatusOff(ctx context.Context, server *hcloud.Serv
 		if time.Now().Before(serverAvailableCondition.LastTransitionTime.Time.Add(serverOffTimeout)) {
 			// Not yet timed out, try again to power on
 			if err := s.scope.HCloudClient.PowerOnServer(ctx, server); err != nil {
-				hcloudutil.HandleRateLimitExceeded(s.scope.HCloudMachine, err, "PowerOnServer")
 				if hcloud.IsError(err, hcloud.ErrorCodeLocked) {
 					// if server is locked, we just retry again
 					return reconcile.Result{RequeueAfter: 30 * time.Second}, nil
 				}
-				return reconcile.Result{}, fmt.Errorf("failed to power on server: %w", err)
+				return reconcile.Result{}, handleRateLimit(s.scope.HCloudMachine, err, "PowerOnServer", "failed to power on server")
 			}
 		} else {
 			// Timed out. Set failure reason
@@ -592,12 +603,11 @@ func (s *Service) handleServerStatusOff(ctx context.Context, server *hcloud.Serv
 	} else {
 		// No condition set yet. Try to power server on.
 		if err := s.scope.HCloudClient.PowerOnServer(ctx, server); err != nil {
-			hcloudutil.HandleRateLimitExceeded(s.scope.HCloudMachine, err, "PowerOnServer")
 			if hcloud.IsError(err, hcloud.ErrorCodeLocked) {
 				// if server is locked, we just retry again
 				return reconcile.Result{RequeueAfter: 30 * time.Second}, nil
 			}
-			return reconcile.Result{}, fmt.Errorf("failed to power on server: %w", err)
+			return reconcile.Result{}, handleRateLimit(s.scope.HCloudMachine, err, "PowerOnServer", "failed to power on server")
 		}
 		conditions.MarkFalse(
 			s.scope.HCloudMachine,
@@ -619,8 +629,7 @@ func (s *Service) handleDeleteServerStatusRunning(ctx context.Context, server *h
 
 	if s.scope.HasServerAvailableCondition() {
 		if err := s.scope.HCloudClient.ShutdownServer(ctx, server); err != nil {
-			hcloudutil.HandleRateLimitExceeded(s.scope.HCloudMachine, err, "ShutdownServer")
-			return reconcile.Result{}, fmt.Errorf("failed to shutdown server: %w", err)
+			return reconcile.Result{}, handleRateLimit(s.scope.HCloudMachine, err, "ShutdownServer", "failed to shutdown server")
 		}
 
 		conditions.MarkFalse(s.scope.HCloudMachine,
@@ -635,9 +644,8 @@ func (s *Service) handleDeleteServerStatusRunning(ctx context.Context, server *h
 
 	// timeout for shutdown has been reached - delete server
 	if err := s.scope.HCloudClient.DeleteServer(ctx, server); err != nil {
-		hcloudutil.HandleRateLimitExceeded(s.scope.HCloudMachine, err, "DeleteServer")
 		record.Warnf(s.scope.HCloudMachine, "FailedDeleteHCloudServer", "Failed to delete HCloud server %s", s.scope.Name())
-		return reconcile.Result{}, fmt.Errorf("failed to delete server: %w", err)
+		return reconcile.Result{}, handleRateLimit(s.scope.HCloudMachine, err, "DeleteServer", "failed to delete server")
 	}
 
 	record.Eventf(s.scope.HCloudMachine, "HCloudServerDeleted", "HCloud server %s deleted", s.scope.Name())
@@ -647,9 +655,8 @@ func (s *Service) handleDeleteServerStatusRunning(ctx context.Context, server *h
 func (s *Service) handleDeleteServerStatusOff(ctx context.Context, server *hcloud.Server) (res reconcile.Result, err error) {
 	// server is off and can be deleted
 	if err := s.scope.HCloudClient.DeleteServer(ctx, server); err != nil {
-		hcloudutil.HandleRateLimitExceeded(s.scope.HCloudMachine, err, "DeleteServer")
 		record.Warnf(s.scope.HCloudMachine, "FailedDeleteHCloudServer", "Failed to delete HCloud server %s", s.scope.Name())
-		return reconcile.Result{}, fmt.Errorf("failed to delete server: %w", err)
+		return reconcile.Result{}, handleRateLimit(s.scope.HCloudMachine, err, "DeleteServer", "failed to delete server")
 	}
 
 	record.Eventf(s.scope.HCloudMachine, "HCloudServerDeleted", "HCloud server %s deleted", s.scope.Name())
@@ -660,12 +667,12 @@ func (s *Service) deleteServerOfLoadBalancer(ctx context.Context, server *hcloud
 	lb := &hcloud.LoadBalancer{ID: s.scope.HetznerCluster.Status.ControlPlaneLoadBalancer.ID}
 
 	if err := s.scope.HCloudClient.DeleteTargetServerOfLoadBalancer(ctx, lb, server); err != nil {
-		hcloudutil.HandleRateLimitExceeded(s.scope.HCloudMachine, err, "DeleteTargetServerOfLoadBalancer")
 		// do not return an error in case the target was not found
 		if strings.Contains(err.Error(), "load_balancer_target_not_found") {
 			return nil
 		}
-		return fmt.Errorf("failed to delete server %s with ID %d as target of load balancer %s with ID %d: %w", server.Name, server.ID, lb.Name, lb.ID, err)
+		errMsg := fmt.Sprintf("failed to delete server %s with ID %d as target of load balancer %s with ID %d", server.Name, server.ID, lb.Name, lb.ID)
+		return handleRateLimit(s.scope.HCloudMachine, err, "DeleteTargetServerOfLoadBalancer", errMsg)
 	}
 	record.Eventf(
 		s.scope.HetznerCluster,
@@ -685,8 +692,8 @@ func (s *Service) findServer(ctx context.Context) (*hcloud.Server, error) {
 	if err == nil {
 		server, err = s.scope.HCloudClient.GetServer(ctx, serverID)
 		if err != nil {
-			hcloudutil.HandleRateLimitExceeded(s.scope.HCloudMachine, err, "GetServer")
-			return nil, fmt.Errorf("failed to get server %d: %w", serverID, err)
+			errMsg := fmt.Sprintf("failed to get server %d", serverID)
+			return nil, handleRateLimit(s.scope.HCloudMachine, err, "GetServer", errMsg)
 		}
 
 		// if server has been found, return it
@@ -702,8 +709,7 @@ func (s *Service) findServer(ctx context.Context) (*hcloud.Server, error) {
 
 	servers, err := s.scope.HCloudClient.ListServers(ctx, opts)
 	if err != nil {
-		hcloudutil.HandleRateLimitExceeded(s.scope.HCloudMachine, err, "ListServers")
-		return nil, fmt.Errorf("failed to list servers: %w", err)
+		return nil, handleRateLimit(s.scope.HCloudMachine, err, "ListServers", "failed to list servers")
 	}
 
 	if len(servers) > 1 {
diff --git a/templates/cluster-templates/bases/hcloud-kcp-ubuntu.yaml b/templates/cluster-templates/bases/hcloud-kcp-ubuntu.yaml
index 64aa8435..9614a3c9 100644
--- a/templates/cluster-templates/bases/hcloud-kcp-ubuntu.yaml
+++ b/templates/cluster-templates/bases/hcloud-kcp-ubuntu.yaml
@@ -166,7 +166,7 @@ spec:
           nameserver 2606:4700:4700::1111
     preKubeadmCommands:
       - set -x
-      - export CONTAINERD=1.7.20 # update: datasource=github-tags depName=containerd/containerd extractVersion=^v(?<version>.*)$ versioning=semver
+      - export CONTAINERD=1.7.21 # update: datasource=github-tags depName=containerd/containerd extractVersion=^v(?<version>.*)$ versioning=semver
       - export KUBERNETES_VERSION=$(echo ${KUBERNETES_VERSION} | sed 's/^v//')
       - export TRIMMED_KUBERNETES_VERSION=$(echo ${KUBERNETES_VERSION} | sed 's/^v//' | awk -F . '{print $1 "." $2}')
       - ARCH=amd64
diff --git a/templates/cluster-templates/bases/hetznerbaremetal-kcp-ubuntu.yaml b/templates/cluster-templates/bases/hetznerbaremetal-kcp-ubuntu.yaml
index 98a110ed..561a1d78 100644
--- a/templates/cluster-templates/bases/hetznerbaremetal-kcp-ubuntu.yaml
+++ b/templates/cluster-templates/bases/hetznerbaremetal-kcp-ubuntu.yaml
@@ -185,7 +185,7 @@ spec:
           nameserver 2606:4700:4700::1111
     preKubeadmCommands:
       - set -x
-      - export CONTAINERD=1.7.20 # update: datasource=github-tags depName=containerd/containerd extractVersion=^v(?<version>.*)$ versioning=semver
+      - export CONTAINERD=1.7.21 # update: datasource=github-tags depName=containerd/containerd extractVersion=^v(?<version>.*)$ versioning=semver
       - export KUBERNETES_VERSION=$(echo ${KUBERNETES_VERSION} | sed 's/^v//')
       - export TRIMMED_KUBERNETES_VERSION=$(echo ${KUBERNETES_VERSION} | sed 's/^v//' | awk -F . '{print $1 "." $2}')
       - localectl set-locale LANG=en_US.UTF-8
diff --git a/templates/cluster-templates/bases/kct-md-0-ubuntu.yaml b/templates/cluster-templates/bases/kct-md-0-ubuntu.yaml
index 8c88b91d..7a785e7e 100644
--- a/templates/cluster-templates/bases/kct-md-0-ubuntu.yaml
+++ b/templates/cluster-templates/bases/kct-md-0-ubuntu.yaml
@@ -55,7 +55,7 @@ spec:
       preKubeadmCommands:
         - set -x
         - grep VERSION= /etc/os-release; uname -a
-        - export CONTAINERD=1.7.20 # update: datasource=github-tags depName=containerd/containerd extractVersion=^v(?<version>.*)$ versioning=semver
+        - export CONTAINERD=1.7.21 # update: datasource=github-tags depName=containerd/containerd extractVersion=^v(?<version>.*)$ versioning=semver
         - export KUBERNETES_VERSION=$(echo ${KUBERNETES_VERSION} | sed 's/^v//')
         - export TRIMMED_KUBERNETES_VERSION=$(echo ${KUBERNETES_VERSION} | sed 's/^v//' | awk -F . '{print $1 "." $2}')
         - ARCH=amd64


Test patch:
diff --git a/api/v1beta1/hetznerbaremetalhost_types_test.go b/api/v1beta1/hetznerbaremetalhost_types_test.go
index 9b7d205f..2f449d2a 100644
--- a/api/v1beta1/hetznerbaremetalhost_types_test.go
+++ b/api/v1beta1/hetznerbaremetalhost_types_test.go
@@ -406,8 +406,6 @@ var _ = Describe("Test ClearRebootAnnotations", func() {
 		expectAnnotations  map[string]string
 	}
 
-	secondRebootAnnotation := RebootAnnotation + "/"
-
 	DescribeTable("Test ClearRebootAnnotations",
 		func(tc testCaseClearRebootAnnotations) {
 			host := HetznerBareMetalHost{}
@@ -417,10 +415,6 @@ var _ = Describe("Test ClearRebootAnnotations", func() {
 
 			Expect(host.Annotations).Should(Equal(tc.expectAnnotations))
 		},
-		Entry("has reboot annotation - one annotation in list", testCaseClearRebootAnnotations{
-			currentAnnotations: map[string]string{secondRebootAnnotation: "reboot", RebootAnnotation: "reboot"},
-			expectAnnotations:  map[string]string{},
-		}),
 		Entry("has multiple reboot annotations - no other annotation in list", testCaseClearRebootAnnotations{
 			currentAnnotations: map[string]string{RebootAnnotation: "reboot"},
 			expectAnnotations:  map[string]string{},
@@ -430,7 +424,7 @@ var _ = Describe("Test ClearRebootAnnotations", func() {
 			expectAnnotations:  map[string]string{"other": "annotation"},
 		}),
 		Entry("has multiple reboot annotations - multiple annotations in list", testCaseClearRebootAnnotations{
-			currentAnnotations: map[string]string{secondRebootAnnotation: "reboot", "other": "annotation", RebootAnnotation: "reboot"},
+			currentAnnotations: map[string]string{"other": "annotation", RebootAnnotation: "reboot"},
 			expectAnnotations:  map[string]string{"other": "annotation"},
 		}),
 		Entry("has no reboot annotation", testCaseClearRebootAnnotations{
@@ -456,7 +450,7 @@ var _ = Describe("Test ClearRebootAnnotations", func() {
 		}),
 		Entry("reboot prefix", testCaseClearRebootAnnotations{
 			annotation: RebootAnnotation + "/" + "suffix",
-			expectBool: true,
+			expectBool: false,
 		}),
 		Entry("other annotation", testCaseClearRebootAnnotations{
 			annotation: "different/annotation",
diff --git a/controllers/controllers_suite_test.go b/controllers/controllers_suite_test.go
index c3459c15..142a20d5 100644
--- a/controllers/controllers_suite_test.go
+++ b/controllers/controllers_suite_test.go
@@ -34,7 +34,6 @@ import (
 	"sigs.k8s.io/controller-runtime/pkg/controller"
 
 	infrav1 "github.com/syself/cluster-api-provider-hetzner/api/v1beta1"
-	hcloudclient "github.com/syself/cluster-api-provider-hetzner/pkg/services/hcloud/client"
 	"github.com/syself/cluster-api-provider-hetzner/test/helpers"
 )
 
@@ -45,10 +44,9 @@ const (
 )
 
 var (
-	testEnv      *helpers.TestEnvironment
-	hcloudClient hcloudclient.Client
-	ctx          = ctrl.SetupSignalHandler()
-	wg           sync.WaitGroup
+	testEnv *helpers.TestEnvironment
+	ctx     = ctrl.SetupSignalHandler()
+	wg      sync.WaitGroup
 
 	defaultPlacementGroupName = "caph-placement-group"
 	defaultFailureDomain      = "fsn1"
@@ -64,7 +62,6 @@ var _ = BeforeSuite(func() {
 	utilruntime.Must(clusterv1.AddToScheme(scheme.Scheme))
 
 	testEnv = helpers.NewTestEnvironment()
-	hcloudClient = testEnv.HCloudClientFactory.NewClient("")
 
 	wg.Add(1)
 
diff --git a/controllers/hcloudmachine_controller_test.go b/controllers/hcloudmachine_controller_test.go
index a6450e2f..c51c7e28 100644
--- a/controllers/hcloudmachine_controller_test.go
+++ b/controllers/hcloudmachine_controller_test.go
@@ -17,9 +17,7 @@ limitations under the License.
 package controllers
 
 import (
-	"fmt"
 	"testing"
-	"time"
 
 	"github.com/hetznercloud/hcloud-go/v2/hcloud"
 	. "github.com/onsi/ginkgo/v2"
@@ -36,6 +34,7 @@ import (
 	"sigs.k8s.io/controller-runtime/pkg/predicate"
 
 	infrav1 "github.com/syself/cluster-api-provider-hetzner/api/v1beta1"
+	hcloudclient "github.com/syself/cluster-api-provider-hetzner/pkg/services/hcloud/client"
 	"github.com/syself/cluster-api-provider-hetzner/pkg/utils"
 )
 
@@ -320,7 +319,10 @@ var _ = Describe("HCloudMachineReconciler", func() {
 
 	Context("Basic test", func() {
 		Context("correct server", func() {
+			var hcloudClient hcloudclient.Client
+
 			BeforeEach(func() {
+				hcloudClient = testEnv.ResetAndGetGlobalHCloudClient()
 				// remove bootstrap infos
 				capiMachine.Spec.Bootstrap = clusterv1.Bootstrap{}
 				Expect(testEnv.Create(ctx, capiMachine)).To(Succeed())
@@ -365,26 +367,23 @@ var _ = Describe("HCloudMachineReconciler", func() {
 				}, timeout).Should(BeTrue())
 			})
 
-			It("creates the HCloud machine in Hetzner 1 (flaky)", func() {
+			It("creates the HCloud machine in Hetzner 1", func() {
 				By("checking that no servers exist")
 
-				Eventually(func(start time.Time) bool {
+				Eventually(func() bool {
 					servers, err := hcloudClient.ListServers(ctx, hcloud.ServerListOpts{
 						ListOpts: hcloud.ListOpts{
 							LabelSelector: utils.LabelsToLabelSelector(map[string]string{hetznerCluster.ClusterTagKey(): "owned"}),
 						},
 					})
 					if err != nil {
-						fmt.Printf("flaky test. ListServers failed: %s\n", err.Error())
 						return false
 					}
 					if len(servers) != 0 {
-						fmt.Printf("flaky test. There are still servers: %+v\n", servers)
 						return false
 					}
-					fmt.Printf("flaky test. OK after %s\n", time.Since(start).String())
 					return true
-				}, 2*timeout, interval).WithArguments(time.Now()).Should(BeTrue())
+				}, timeout, interval).Should(BeTrue())
 
 				By("checking that bootstrap condition is not ready")
 
@@ -517,7 +516,10 @@ var _ = Describe("HCloudMachineReconciler", func() {
 		})
 
 		Context("without network", func() {
+			var hcloudClient hcloudclient.Client
+
 			BeforeEach(func() {
+				hcloudClient = testEnv.ResetAndGetGlobalHCloudClient()
 				hetznerCluster.Spec.HCloudNetwork.Enabled = false
 				Expect(testEnv.Create(ctx, hetznerCluster)).To(Succeed())
 				Expect(testEnv.Create(ctx, hcloudMachine)).To(Succeed())
@@ -543,7 +545,10 @@ var _ = Describe("HCloudMachineReconciler", func() {
 		})
 
 		Context("without placement groups", func() {
+			var hcloudClient hcloudclient.Client
+
 			BeforeEach(func() {
+				hcloudClient = testEnv.ResetAndGetGlobalHCloudClient()
 				hetznerCluster.Spec.HCloudPlacementGroups = nil
 				Expect(testEnv.Create(ctx, hetznerCluster)).To(Succeed())
 
@@ -609,7 +614,9 @@ var _ = Describe("HCloudMachineReconciler", func() {
 		})
 
 		Context("with public network specs", func() {
+			var hcloudClient hcloudclient.Client
 			BeforeEach(func() {
+				hcloudClient = testEnv.ResetAndGetGlobalHCloudClient()
 				hcloudMachine.Spec.PublicNetwork = &infrav1.PublicNetworkSpec{
 					EnableIPv4: false,
 					EnableIPv6: false,
diff --git a/controllers/hcloudremediation_controller_test.go b/controllers/hcloudremediation_controller_test.go
index c1910e11..7fee5986 100644
--- a/controllers/hcloudremediation_controller_test.go
+++ b/controllers/hcloudremediation_controller_test.go
@@ -30,6 +30,7 @@ import (
 	"sigs.k8s.io/controller-runtime/pkg/client"
 
 	infrav1 "github.com/syself/cluster-api-provider-hetzner/api/v1beta1"
+	hcloudclient "github.com/syself/cluster-api-provider-hetzner/pkg/services/hcloud/client"
 	hcloudutil "github.com/syself/cluster-api-provider-hetzner/pkg/services/hcloud/util"
 	"github.com/syself/cluster-api-provider-hetzner/pkg/utils"
 )
@@ -186,6 +187,11 @@ var _ = Describe("HCloudRemediationReconciler", func() {
 	})
 
 	Context("Basic test", func() {
+		var hcloudClient hcloudclient.Client
+		BeforeEach(func() {
+			hcloudClient = testEnv.ResetAndGetGlobalHCloudClient()
+		})
+
 		It("creates the hcloudRemediation successfully", func() {
 			Expect(testEnv.Create(ctx, hcloudRemediation)).To(Succeed())
 
diff --git a/controllers/hetznerbaremetalremediation_controller_test.go b/controllers/hetznerbaremetalremediation_controller_test.go
index fd2aa54f..7950cb4c 100644
--- a/controllers/hetznerbaremetalremediation_controller_test.go
+++ b/controllers/hetznerbaremetalremediation_controller_test.go
@@ -252,12 +252,12 @@ var _ = Describe("HetznerBareMetalRemediationReconciler", func() {
 				Expect(testEnv.Cleanup(ctx, testNs, hetznerSecret, osSSHSecret, rescueSSHSecret, bootstrapSecret)).To(Succeed())
 			})
 
-			Context("HetznerBaremetalHost doesn't exist", func() {
+			Context("HetznerBareMetalHost doesn't exist", func() {
 				AfterEach(func() {
 					Expect(testEnv.Cleanup(ctx, hetznerBareMetalRemediation, hetznerBaremetalMachine)).To(Succeed())
 				})
 
-				It("should not remediate if no annotations is present in the hetznerBaremetalMachine (flaky)", func() {
+				It("should not remediate if no annotations is present in the hetznerBaremetalMachine", func() {
 					Expect(testEnv.Create(ctx, hetznerBaremetalMachine)).To(Succeed())
 					Expect(testEnv.Create(ctx, hetznerBareMetalRemediation)).To(Succeed())
 
@@ -287,7 +287,7 @@ var _ = Describe("HetznerBareMetalRemediationReconciler", func() {
 				})
 			})
 
-			Context("HetznerBaremetalHost exist", func() {
+			Context("HetznerBareMetalHost exist", func() {
 				BeforeEach(func() {
 					hostKey = client.ObjectKey{Name: hostName, Namespace: testNs.Name}
 
diff --git a/controllers/hetznercluster_controller_test.go b/controllers/hetznercluster_controller_test.go
index 1a1cce88..c60a7053 100644
--- a/controllers/hetznercluster_controller_test.go
+++ b/controllers/hetznercluster_controller_test.go
@@ -35,6 +35,7 @@ import (
 	"sigs.k8s.io/controller-runtime/pkg/event"
 
 	infrav1 "github.com/syself/cluster-api-provider-hetzner/api/v1beta1"
+	hcloudclient "github.com/syself/cluster-api-provider-hetzner/pkg/services/hcloud/client"
 	"github.com/syself/cluster-api-provider-hetzner/pkg/utils"
 	"github.com/syself/cluster-api-provider-hetzner/test/helpers"
 )
@@ -333,6 +334,12 @@ var _ = Describe("Hetzner ClusterReconciler", func() {
 		})
 
 		Context("load balancer", func() {
+			var hcloudClient hcloudclient.Client
+
+			BeforeEach(func() {
+				hcloudClient = testEnv.ResetAndGetGlobalHCloudClient()
+			})
+
 			It("should create load balancer and update it accordingly", func() {
 				Expect(testEnv.Create(ctx, instance)).To(Succeed())
 
@@ -735,8 +742,10 @@ var _ = Describe("Hetzner ClusterReconciler", func() {
 
 		Context("HetznerMachines belonging to the cluster", func() {
 			var bootstrapSecret *corev1.Secret
+			var hcloudClient hcloudclient.Client
 
 			BeforeEach(func() {
+				hcloudClient = testEnv.ResetAndGetGlobalHCloudClient()
 				bootstrapSecret = getDefaultBootstrapSecret(namespace)
 				Expect(testEnv.Create(ctx, bootstrapSecret)).To(Succeed())
 			})
@@ -773,8 +782,10 @@ var _ = Describe("Hetzner ClusterReconciler", func() {
 
 		Context("Placement groups", func() {
 			var bootstrapSecret *corev1.Secret
+			var hcloudClient hcloudclient.Client
 
 			BeforeEach(func() {
+				hcloudClient = testEnv.ResetAndGetGlobalHCloudClient()
 				// Create the bootstrap secret
 				bootstrapSecret = getDefaultBootstrapSecret(namespace)
 				Expect(testEnv.Create(ctx, bootstrapSecret)).To(Succeed())
diff --git a/pkg/services/hcloud/client/fake/hcloud_client_test.go b/pkg/services/hcloud/client/fake/hcloud_client_test.go
index f9e13c15..55058599 100644
--- a/pkg/services/hcloud/client/fake/hcloud_client_test.go
+++ b/pkg/services/hcloud/client/fake/hcloud_client_test.go
@@ -25,6 +25,7 @@ import (
 	. "github.com/onsi/ginkgo/v2"
 	. "github.com/onsi/gomega"
 
+	hcloudclient "github.com/syself/cluster-api-provider-hetzner/pkg/services/hcloud/client"
 	"github.com/syself/cluster-api-provider-hetzner/pkg/services/hcloud/client/fake"
 )
 
@@ -75,12 +76,13 @@ var _ = Describe("Load balancer", func() {
 		Subnets: []hcloud.NetworkSubnet{{IPRange: &net.IPNet{IP: net.IP("1.2.3.6")}}},
 	}
 
-	client := factory.NewClient("")
+	var client hcloudclient.Client
 
 	var lb *hcloud.LoadBalancer
 	var network *hcloud.Network
 
 	BeforeEach(func() {
+		client = factory.NewClient("").Reset()
 		_, err := client.CreateLoadBalancer(ctx, opts)
 		Expect(err).To(Succeed())
 
@@ -92,9 +94,6 @@ var _ = Describe("Load balancer", func() {
 		network, err = client.CreateNetwork(ctx, networkOpts)
 		Expect(err).To(Succeed())
 	})
-	AfterEach(func() {
-		client.Close()
-	})
 
 	It("creates a load balancer and returns it with an ID", func() {
 		Expect(lb.ID).ToNot(Equal(0))
@@ -421,18 +420,21 @@ var _ = Describe("Load balancer", func() {
 		err := client.DeleteServiceFromLoadBalancer(ctx, lb, 999)
 		Expect(err).ToNot(Succeed())
 	})
+})
 
-	_ = Describe("Images", func() {
-		var listOpts hcloud.ImageListOpts
-		client := factory.NewClient("")
-		BeforeEach(func() {
-			listOpts.LabelSelector = "caph-image-name==fedora-control-plane"
-		})
-		It("lists at least one image", func() {
-			resp, err := client.ListImages(ctx, listOpts)
-			Expect(err).To(Succeed())
-			Expect(len(resp)).To(BeNumerically(">", 0))
-		})
+var _ = Describe("Images", func() {
+	var listOpts hcloud.ImageListOpts
+	var client hcloudclient.Client
+
+	BeforeEach(func() {
+		client = factory.NewClient("").Reset()
+
+		listOpts.LabelSelector = "caph-image-name==fedora-control-plane"
+	})
+	It("lists at least one image", func() {
+		resp, err := client.ListImages(ctx, listOpts)
+		Expect(err).To(Succeed())
+		Expect(len(resp)).To(BeNumerically(">", 0))
 	})
 })
 
@@ -440,7 +442,7 @@ var _ = Describe("Server", func() {
 	var listOpts hcloud.ServerListOpts
 	listOpts.LabelSelector = labelSelector
 
-	client := factory.NewClient("")
+	var client hcloudclient.Client
 
 	opts := hcloud.ServerCreateOpts{
 		Name: "test-server",
@@ -464,6 +466,8 @@ var _ = Describe("Server", func() {
 
 	BeforeEach(func() {
 		var err error
+		client = factory.NewClient("").Reset()
+
 		server, err = client.CreateServer(ctx, opts)
 		Expect(err).To(Succeed())
 
@@ -478,9 +482,6 @@ var _ = Describe("Server", func() {
 		})
 		Expect(err).To(Succeed())
 	})
-	AfterEach(func() {
-		client.Close()
-	})
 
 	It("creates a server with an ID", func() {
 		Expect(server.ID).ToNot(Equal(0))
@@ -609,7 +610,7 @@ var _ = Describe("Network", func() {
 	var listOpts hcloud.NetworkListOpts
 	listOpts.LabelSelector = labelSelector
 
-	client := factory.NewClient("")
+	var client hcloudclient.Client
 
 	opts := hcloud.NetworkCreateOpts{
 		Name: "test-network",
@@ -625,12 +626,10 @@ var _ = Describe("Network", func() {
 
 	BeforeEach(func() {
 		var err error
+		client = factory.NewClient("").Reset()
 		network, err = client.CreateNetwork(ctx, opts)
 		Expect(err).To(Succeed())
 	})
-	AfterEach(func() {
-		client.Close()
-	})
 
 	It("creates a network with an ID", func() {
 		Expect(network.ID).ToNot(Equal(0))
@@ -675,13 +674,13 @@ var _ = Describe("Placement groups", func() {
 		Type: "stream",
 	}
 
-	client := factory.NewClient("")
-
+	var client hcloudclient.Client
 	var server *hcloud.Server
 	var placementGroup *hcloud.PlacementGroup
 
 	BeforeEach(func() {
 		var err error
+		client = factory.NewClient("").Reset()
 		server, err = client.CreateServer(ctx, hcloud.ServerCreateOpts{
 			Name: "test-server",
 			Labels: map[string]string{
@@ -703,9 +702,6 @@ var _ = Describe("Placement groups", func() {
 		placementGroup, err = client.CreatePlacementGroup(ctx, opts)
 		Expect(err).To(Succeed())
 	})
-	AfterEach(func() {
-		client.Close()
-	})
 
 	It("creates a placementGroup with an ID", func() {
 		Expect(placementGroup.ID).ToNot(Equal(0))
diff --git a/pkg/services/hcloud/server/server_test.go b/pkg/services/hcloud/server/server_test.go
index d7e2fc33..8f8d89fa 100644
--- a/pkg/services/hcloud/server/server_test.go
+++ b/pkg/services/hcloud/server/server_test.go
@@ -18,11 +18,13 @@ package server
 
 import (
 	"context"
+	"fmt"
 	"time"
 
 	"github.com/hetznercloud/hcloud-go/v2/hcloud"
 	. "github.com/onsi/ginkgo/v2"
 	. "github.com/onsi/gomega"
+	corev1 "k8s.io/api/core/v1"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 	"k8s.io/utils/ptr"
 	clusterv1 "sigs.k8s.io/cluster-api/api/v1beta1"
@@ -339,3 +341,105 @@ var _ = Describe("Test ValidateLabels", func() {
 		}),
 	)
 })
+
+var _ = Describe("Test handleRateLimit", func() {
+	type testCaseHandleRateLimit struct {
+		hm              *infrav1.HCloudMachine
+		err             error
+		functionName    string
+		errMsg          string
+		expectError     error
+		expectCondition bool
+	}
+
+	DescribeTable("Test handleRateLimit",
+		func(tc testCaseHandleRateLimit) {
+			err := handleRateLimit(tc.hm, tc.err, tc.functionName, tc.errMsg)
+			if tc.expectError != nil {
+				Expect(err).To(MatchError(tc.expectError))
+			} else {
+				Expect(err).To(BeNil())
+			}
+			if tc.expectCondition {
+				Expect(isPresentAndFalseWithReason(tc.hm, infrav1.HetznerAPIReachableCondition, infrav1.RateLimitExceededReason)).To(BeTrue())
+			} else {
+				Expect(conditions.Get(tc.hm, infrav1.HetznerAPIReachableCondition)).To(BeNil())
+			}
+		},
+		Entry("machine not ready, rate limit exceeded error", testCaseHandleRateLimit{
+			hm: &infrav1.HCloudMachine{
+				Status: infrav1.HCloudMachineStatus{Ready: false},
+			},
+			err:             hcloud.Error{Code: hcloud.ErrorCodeRateLimitExceeded},
+			functionName:    "TestFunction",
+			errMsg:          "Test error message",
+			expectError:     fmt.Errorf("Test error message: %w", hcloud.Error{Code: hcloud.ErrorCodeRateLimitExceeded}),
+			expectCondition: true,
+		}),
+		Entry("machine has deletion timestamp, rate limit exceeded error", testCaseHandleRateLimit{
+			hm: &infrav1.HCloudMachine{
+				ObjectMeta: metav1.ObjectMeta{
+					DeletionTimestamp: &metav1.Time{Time: time.Now()},
+				},
+				Status: infrav1.HCloudMachineStatus{Ready: true},
+			},
+			err:             hcloud.Error{Code: hcloud.ErrorCodeRateLimitExceeded},
+			functionName:    "TestFunction",
+			errMsg:          "Test error message",
+			expectError:     fmt.Errorf("Test error message: %w", hcloud.Error{Code: hcloud.ErrorCodeRateLimitExceeded}),
+			expectCondition: true,
+		}),
+		Entry("machine not ready, has deletion timestamp, rate limit exceeded error", testCaseHandleRateLimit{
+			hm: &infrav1.HCloudMachine{
+				ObjectMeta: metav1.ObjectMeta{
+					DeletionTimestamp: &metav1.Time{Time: time.Now()},
+				},
+				Status: infrav1.HCloudMachineStatus{Ready: false},
+			},
+			err:             hcloud.Error{Code: hcloud.ErrorCodeRateLimitExceeded},
+			functionName:    "TestFunction",
+			errMsg:          "Test error message",
+			expectError:     fmt.Errorf("Test error message: %w", hcloud.Error{Code: hcloud.ErrorCodeRateLimitExceeded}),
+			expectCondition: true,
+		}),
+		Entry("machine ready, rate limit exceeded error", testCaseHandleRateLimit{
+			hm: &infrav1.HCloudMachine{
+				Status: infrav1.HCloudMachineStatus{Ready: true},
+			},
+			err:             hcloud.Error{Code: hcloud.ErrorCodeRateLimitExceeded},
+			functionName:    "TestFunction",
+			errMsg:          "Test error message",
+			expectError:     nil,
+			expectCondition: false,
+		}),
+		Entry("machine ready, other error", testCaseHandleRateLimit{
+			hm: &infrav1.HCloudMachine{
+				Status: infrav1.HCloudMachineStatus{Ready: true},
+			},
+			err:             hcloud.Error{Code: hcloud.ErrorCodeResourceUnavailable},
+			functionName:    "TestFunction",
+			errMsg:          "Test error message",
+			expectError:     fmt.Errorf("Test error message: %w", hcloud.Error{Code: hcloud.ErrorCodeResourceUnavailable}),
+			expectCondition: false,
+		}),
+		Entry("machine not ready, other error", testCaseHandleRateLimit{
+			hm: &infrav1.HCloudMachine{
+				Status: infrav1.HCloudMachineStatus{Ready: false},
+			},
+			err:             hcloud.Error{Code: hcloud.ErrorCodeConflict},
+			functionName:    "TestFunction",
+			errMsg:          "Test conflict error message",
+			expectError:     fmt.Errorf("Test conflict error message: %w", hcloud.Error{Code: hcloud.ErrorCodeConflict}),
+			expectCondition: false,
+		}),
+	)
+})
+
+func isPresentAndFalseWithReason(getter conditions.Getter, condition clusterv1.ConditionType, reason string) bool {
+	if !conditions.Has(getter, condition) {
+		return false
+	}
+	objectCondition := conditions.Get(getter, condition)
+	return objectCondition.Status == corev1.ConditionFalse &&
+		objectCondition.Reason == reason
+}
diff --git a/test/helpers/envtest.go b/test/helpers/envtest.go
index d6c22587..804da7cd 100644
--- a/test/helpers/envtest.go
+++ b/test/helpers/envtest.go
@@ -257,6 +257,11 @@ func (t *TestEnvironment) CreateKubeconfigSecret(ctx context.Context, cluster *c
 	return t.Create(ctx, kubeconfig.GenerateSecret(cluster, kubeconfig.FromEnvTestConfig(t.Config, cluster)))
 }
 
+// ResetAndGetGlobalHCloudClient resets the cache of the fake Client and returns it.
+func (t *TestEnvironment) ResetAndGetGlobalHCloudClient() hcloudclient.Client {
+	return t.HCloudClientFactory.NewClient("").Reset()
+}
+
 func getFilePathToCAPICRDs(root string) string {
 	mod, err := newMod(filepath.Join(root, "go.mod"))
 	if err != nil {