Repository: kubernetes-sigs/cluster-api-provider-aws

Issue:
ELB health check fails with Kubernetes >=v1.30.x
/kind bug

**What steps did you take and what happened:**
Follow the [quickstart documentation ](https://cluster-api.sigs.k8s.io/user/quick-start.html) with Kubernetes v1.30.5 and a custom built AMI (the public AMIs are missing for that version and the default v1.31.0 version).
The ELB Health Check fails and the cluster is stuck after creating the first control-plane instance. The AWS console shows that 0 of 1 instanced are in service.

* The CAPA API defaults [create a Classic ELB](https://github.com/kubernetes-sigs/cluster-api-provider-aws/blob/1245ff7a81a552a2767d21ed1128678656c24042/api/v1beta2/defaults.go#L70) with an [SSL health check target](https://github.com/kubernetes-sigs/cluster-api-provider-aws/blob/1245ff7a81a552a2767d21ed1128678656c24042/pkg/cloud/services/elb/loadbalancer.go#L1724). (HTTPS also doesn't work, but TCP does) 
* Starting in Go 1.22 the RSA ciphers were removed - https://github.com/golang/go/issues/63413
* Kubernetes >v1.30 switched to Go 1.22 in recent releases https://github.com/kubernetes/kubernetes/commit/ddb0b8da9dbcd3d507001c5cf8a01708816f3cd5


**What did you expect to happen:**
The defaults should result in a working cluster.

**Anything else you would like to add:**
1. Changing the health check to TCP in the AWS console did fix the check, but this update is not allowed by a webhook here and even after removing the webhook, the new value from AWSCluster never got updated.

2. Setting this on the apiserver and other control-plane components allowed the ELB health check to pass
```
tls-cipher-suites: ...,TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_3DES_EDE_CBC_SHA
```
3. Using an NLB loadbalancer works
```
  controlPlaneLoadBalancer:
    loadBalancerType: nlb
```

Some discussion about this in the Kuberentes slack https://kubernetes.slack.com/archives/C3QUFP0QM/p1726622974749509

**Environment:**

- Cluster-api-provider-aws version: 2.6.1
- Kubernetes version: (use `kubectl version`): v1.30.5
- OS (e.g. from `/etc/os-release`):  

Discussion:
@AndiDog: /triage accepted
/priority important-soon@richardcase: I'm running into this as well.@richardcase: /priority critical-urgent
/milestone v2.7.0@richardcase: This was discussed at the office hours 14th October 2024. The summary is that:

- We should update the templates, docs (and at releases) notes on how to explicitly specify the tls cipher suites to ythe kube components. Also, recommend that new clusters consider using nlb instead of classic elb.
- In the future with an API version bump we could consider making nlb the default. @richardcase: /help@k8s-ci-robot: @richardcase: 
	This request has been marked as needing help from a contributor.

### Guidelines
Please ensure that the issue body includes answers to the following questions:
- Why are we solving this issue?
- To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points?
- Does this issue have zero to low barrier of entry?
- How can the assignee reach out to you for help?


For more details on the requirements of such an issue, please see [here](https://git.k8s.io/community/contributors/guide/help-wanted.md) and ensure that they are met.

If this request no longer meets these requirements, the label can be removed
by commenting with the `/remove-help` command.


<details>

In response to [this](https://github.com/kubernetes-sigs/cluster-api-provider-aws/issues/5139):

>/help


Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.
</details>@richardcase: /milestone v2.8.0@richardcase: /milestone v2.7.2@richardcase: I tried setting the tls cipher suites but that didn't work:

https://gist.github.com/richardcase/47118a404bc832904c399ba1360462f2@dkoshkin: @richardcase I wasn't able to just apply your spec directly because of some IAM issues, but was able to create by explicitly setting this public AMI:

```
spec:
  ami:
    id: ami-0797a44e8719c5e53
```

AWSCluster
```
  controlPlaneLoadBalancer:
    crossZoneLoadBalancing: false
    healthCheckProtocol: HTTPS
    loadBalancerType: classic
    scheme: internet-facing
```

and KCP:
```
    initConfiguration:
      localAPIEndpoint: {}
      nodeRegistration:
        imagePullPolicy: IfNotPresent
        kubeletExtraArgs:
          cloud-provider: external
          tls-cipher-suites: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256,TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_3DES_EDE_CBC_SHA
        name: '{{ ds.meta_data.local_hostname }}'
    joinConfiguration:
      discovery: {}
      nodeRegistration:
        imagePullPolicy: IfNotPresent
        kubeletExtraArgs:
          cloud-provider: external
          tls-cipher-suites: TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256,TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_3DES_EDE_CBC_SHA
        name: '{{ ds.meta_data.local_hostname }}'
```@AndiDog: I got it working with this additional argument in the template (I'm using `cluster-template-flatcar-machinepool.yaml` but that shouldn't matter):

```
apiVersion: controlplane.cluster.x-k8s.io/v1beta1
kind: KubeadmControlPlane
metadata:
  name: "${CLUSTER_NAME}-control-plane"
spec:
  kubeadmConfigSpec:
    clusterConfiguration:
      apiServer:
        extraArgs:
          cloud-provider: external

          # This is needed for Kubernetes v1.30+ since else it uses the Go defaults which don't
          # work with AWS classic load balancers, see
          # https://github.com/kubernetes-sigs/cluster-api-provider-aws/issues/5139. If you use
          # another load balancer type such as NLB, this is not needed.
          #
          # The list consists of the secure ciphers from Go 1.23.3, plus some less secure
          # RSA ciphers which the AWS classic load balancer instance health check supports.
          tls-cipher-suites: TLS_AES_128_GCM_SHA256,TLS_AES_256_GCM_SHA384,TLS_CHACHA20_POLY1305_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256,TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA
```

Do we really want to hardcode these less secure settings in the template? This makes it very likely for users to blindly take it over. I'm rather thinking of other options:

- Explicitly set NLB as load balancer type in templates, making it easier for users to make the switch later once we make breaking changes to the API specs. However, these template changes should also go into new 2.x release, not a patch release. Document what needs to be done for ELBs. This gives some chance of silent breakage for users (different LB type than expected; what happens if they apply the changed manifest on existing clusters...).
- Only create a documentation page about ELBs. Hard to figure out for users what's wrong.
- (My preference:) Perform the correction in CAPA code with an `if` block. This is the easiest way to make the defaults work immediately without changes by the users, and also it's easy to rip it out later if AWS should improve TLS cipher support in ELB instance health checks.@AndiDog: We talked in the office hours to consider switching the default type to NLB.

Next steps:

- Test what happens when the LB type changes (downtime?). The webhook code currently doesn't forbid a change of that field.
- Test use of `secondaryControlPlaneLoadBalancer` or other alternatives to do a migration
@k8s-triage-robot: This issue is labeled with `priority/critical-urgent` but has not been updated in over 30 days, and should be re-triaged.
Critical-urgent issues must be actively worked on as someone's top priority right now.

You can:
- Confirm that this issue is still relevant with `/triage accepted` (org members only)
- Deprioritize it with `/priority {important-soon, important-longterm, backlog}`
- Close this issue with `/close`

For more details on the triage process, see https://www.kubernetes.dev/docs/guide/issue-triage/

/remove-triage accepted@yoctozepto: > Next steps:
> 
> * Test what happens when the LB type changes (downtime?). The webhook code currently doesn't forbid a change of that field.

The reconciler fails afterwards. Looks like a change in the loadbalancer naming scheme has happened as well (`test-apiserver` vs `default-test-apiserver`):

```
	could not register machine to load balancer: could not register control plane instance "i-03312db3fdf64ffea" with load balancer - error determining registration status: error describing ELB "default-test-apiserver": LoadBalancerNotFound: Load balancers '[default-test-apiserver]' not found
		status code: 400, request id: dffbc514-8563-4761-980d-d1cbfad3a6e7
 >
E0116 13:03:59.918681       1 controller.go:324] "Reconciler error" err=<
	could not register machine to load balancer: could not register control plane instance "i-03312db3fdf64ffea" with load balancer - error determining registration status: error describing ELB "default-test-apiserver": LoadBalancerNotFound: Load balancers '[default-test-apiserver]' not found
		status code: 400, request id: dffbc514-8563-4761-980d-d1cbfad3a6e7
 > controller="awsmachine" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="AWSMachine" AWSMachine="default/test-control-plane-2dcjf" namespace="default" name="test-control-plane-2dcjf" reconcileID="1a2929bf-1ca0-49e9-af28-4d9394c84718"
E0116 13:04:00.700097       1 awsmachine_controller.go:628] "failed to reconcile LB attachment" err=<
	could not register machine to load balancer: could not register control plane instance "i-03312db3fdf64ffea" with load balancer - error determining registration status: error describing ELB "default-test-apiserver": LoadBalancerNotFound: Load balancers '[default-test-apiserver]' not found
		status code: 400, request id: ce6b72c5-f4bf-4720-bf3f-125b1549dac3
 >
E0116 13:04:00.705525       1 controller.go:324] "Reconciler error" err=<
	could not register machine to load balancer: could not register control plane instance "i-03312db3fdf64ffea" with load balancer - error determining registration status: error describing ELB "default-test-apiserver": LoadBalancerNotFound: Load balancers '[default-test-apiserver]' not found
		status code: 400, request id: ce6b72c5-f4bf-4720-bf3f-125b1549dac3
 > controller="awsmachine" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="AWSMachine" AWSMachine="default/test-control-plane-2dcjf" namespace="default" name="test-control-plane-2dcjf" reconcileID="a4ded9f9-337f-49d6-aa8f-4a0c33305763"
E0116 13:04:01.003343       1 awscluster_controller.go:279] "failed to reconcile load balancer" err="no loadbalancer exists for the AWSCluster test, the cluster has become unrecoverable and should be deleted manually: no load balancer found with name: \"default-test-apiserver\"" controller="awscluster" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="AWSCluster" AWSCluster="default/test" namespace="default" name="test" reconcileID="3b022292-e663-444c-b59c-945484e1649b" cluster="default/test"
E0116 13:04:01.040211       1 controller.go:324] "Reconciler error" err="no loadbalancer exists for the AWSCluster test, the cluster has become unrecoverable and should be deleted manually: no load balancer found with name: \"default-test-apiserver\"" controller="awscluster" controllerGroup="infrastructure.cluster.x-k8s.io" controllerKind="AWSCluster" AWSCluster="default/test" namespace="default" name="test" reconcileID="3b022292-e663-444c-b59c-945484e1649b"
```

Even if it did work, it would break existing clients as there is no way to convert ELB, just create a new one and then manually (or with the help of DNS, think Route 53) migrate the existing clients.

---

> Test use of secondaryControlPlaneLoadBalancer or other alternatives to do a migration

This looks interesting as the enabler of the proper way forward in my opinion (read on).

---

Since the working (NLB) alternative uses the TCP healthcheck by default - why not make the default classic ELB use TCP healthchecks by default as a current workaround? I can agree that the verification of SSL/TLS is good beyond just TCP but still not as comprehensive as proper HTTPS healthcheck and so this default change does not really dumb down the defaults much in my opinion.

Note there are no raw SSL/TLS checks in non-classic ELBs.

---

I see the way forward would be to migrate to NLB with HTTPS checks as they should be the most comprehensive (and working), i.e.:

```
spec:
  controlPlaneLoadBalancer:
    loadBalancerType: nlb
    healthCheckProtocol: HTTPS
```

See also notes below.

---

Other notes:

* switching from NLB to ALB in place is not possible either [expected]
* **another bug**: ALB fails to reconcile because the code creates target groups only with TCP protocol
  * what is worse, it spams new and new target groups - I ended up with 200+ (!) after a while of leaving it running - it did not clean them up after removing the cluster either
* HTTPS works as the `healthCheckProtocol` for NLB (apart from TCP in case of classic and NLB)
  * note though that `healthCheckProtocol` is marked immutable so this needs a deletion and recreation of the cluster resource
    * but there is nothing on AWS side that mandates having this immutable - lifting this limitation would help with making switches
  * *a minor bug*: the HTTPS healthcheck seems to be accepting way too many codes as success, it should only really accept 200@richardcase: /milestone v2.8@nrb: Talking with @damdo some, would it work to update the cipher suites for v2.8, then change the default in v2.9? This gives users some time to prepare for the change in default.

Another option could be to use TCP health checks instead of updating the cipher suites, but I think either way, the default will need to change by v2.9 and we'll need to provide either automatic support for it, or well-tested documentation on how to change over.@nrb: #5338 updated a test case that was breaking.

Talking to Richard and Damiano, we think this is how we're going to approach the problem:

Starting in v2.8.0 for new installations, we'll update all the templates to use Network Load Balancers. We will also be marking the classic load balancer default as deprecated. This will start a 9 month/3 version timer until the default is updated within the API to be a Network Load Balancer.

For existing clusters that are using the defaults, the following logic will apply:
  * If the load balancer is classic AND ((No protocol is set OR the protocol is SSL) AND no other fields have changed)
  ** Set the protocol to TCP
  * https://github.com/kubernetes-sigs/cluster-api-provider-aws/blob/f36e8d87e8749562648331ff4d81aed96c0e0b34/api/v1beta2/awscluster_webhook.go#L187 is roughly where that will happen, and the logic will need to be tweaked to allow this mutation.
  * This will need an extended e2e test that goes from Kubernetes 1.29/CAPA 2.7 to Kubernetes 1.30/CAPA 2.8+ in order to validate the behavior.

For existing clusters that are _not_ using the defaults, error and inform the user that they'll need to migrate. The webhook will need to be update to allow the user to do the change themselves.

@richardcase: I just tested automatically changing the health check protocol and this works nicely with minimal code changes. I will create a PR with the changes after a bit more testing.@richardcase: /assign
/lifecycle active@damdo: I put up:
- https://github.com/kubernetes-sigs/cluster-api-provider-aws/pull/5338 to update e2e testing to use NLBs
- https://github.com/kubernetes-sigs/cluster-api-provider-aws/pull/5345 to update CAPA templates to use NLBs

Solution patch:
diff --git a/templates/cluster-template-flatcar.yaml b/templates/cluster-template-flatcar.yaml
index 3084ed88b..f8de3fc55 100644
--- a/templates/cluster-template-flatcar.yaml
+++ b/templates/cluster-template-flatcar.yaml
@@ -26,6 +26,9 @@ metadata:
 spec:
   region: "${AWS_REGION}"
   sshKeyName: "${AWS_SSH_KEY_NAME}"
+  controlPlaneLoadBalancer:
+    loadBalancerType: nlb
+    healthCheckProtocol: HTTPS
   s3Bucket:
     controlPlaneIAMInstanceProfile: control-plane.cluster-api-provider-aws.sigs.k8s.io
     name: "${AWS_S3_BUCKET_NAME}"
diff --git a/templates/cluster-template-machinepool.yaml b/templates/cluster-template-machinepool.yaml
index 2b4b74dc0..253de2cd7 100644
--- a/templates/cluster-template-machinepool.yaml
+++ b/templates/cluster-template-machinepool.yaml
@@ -24,6 +24,9 @@ kind: AWSCluster
 metadata:
   name: "${CLUSTER_NAME}"
 spec:
+  controlPlaneLoadBalancer:
+    loadBalancerType: nlb
+    healthCheckProtocol: HTTPS
   region: "${AWS_REGION}"
   sshKeyName: "${AWS_SSH_KEY_NAME}"
 ---
diff --git a/templates/cluster-template-multitenancy-clusterclass.yaml b/templates/cluster-template-multitenancy-clusterclass.yaml
index 73c9ee8e2..a7599b0fb 100644
--- a/templates/cluster-template-multitenancy-clusterclass.yaml
+++ b/templates/cluster-template-multitenancy-clusterclass.yaml
@@ -150,7 +150,10 @@ metadata:
   name: multi-tenancy
 spec:
   template:
-    spec: {}
+    spec:
+      controlPlaneLoadBalancer:
+        loadBalancerType: nlb
+        healthCheckProtocol: HTTPS
 ---
 kind: KubeadmControlPlaneTemplate
 apiVersion: controlplane.cluster.x-k8s.io/v1beta1
diff --git a/templates/cluster-template-simple-clusterclass.yaml b/templates/cluster-template-simple-clusterclass.yaml
index c0a52ee39..af0956294 100644
--- a/templates/cluster-template-simple-clusterclass.yaml
+++ b/templates/cluster-template-simple-clusterclass.yaml
@@ -159,7 +159,10 @@ metadata:
   name: quick-start
 spec:
   template:
-    spec: { }
+    spec:
+      controlPlaneLoadBalancer:
+        loadBalancerType: nlb
+        healthCheckProtocol: HTTPS
 ---
 kind: KubeadmControlPlaneTemplate
 apiVersion: controlplane.cluster.x-k8s.io/v1beta1
diff --git a/templates/cluster-template.yaml b/templates/cluster-template.yaml
index 0ad3aa6d5..35636f203 100644
--- a/templates/cluster-template.yaml
+++ b/templates/cluster-template.yaml
@@ -24,6 +24,9 @@ kind: AWSCluster
 metadata:
   name: "${CLUSTER_NAME}"
 spec:
+  controlPlaneLoadBalancer:
+    loadBalancerType: nlb
+    healthCheckProtocol: HTTPS
   region: "${AWS_REGION}"
   sshKeyName: "${AWS_SSH_KEY_NAME}"
 ---



Test patch:
diff --git a/pkg/cloud/services/eks/oidc_test.go b/pkg/cloud/services/eks/oidc_test.go
index b86488635..ffc6c4446 100644
--- a/pkg/cloud/services/eks/oidc_test.go
+++ b/pkg/cloud/services/eks/oidc_test.go
@@ -17,6 +17,11 @@ limitations under the License.
 package eks
 
 import (
+	"crypto/sha1"
+	"crypto/tls"
+	"crypto/x509"
+	"encoding/hex"
+	"encoding/pem"
 	"net/http"
 	"net/http/httptest"
 	"testing"
@@ -35,10 +40,13 @@ import (
 	ekscontrolplanev1 "sigs.k8s.io/cluster-api-provider-aws/v2/controlplane/eks/api/v1beta2"
 	"sigs.k8s.io/cluster-api-provider-aws/v2/pkg/cloud/scope"
 	"sigs.k8s.io/cluster-api-provider-aws/v2/pkg/cloud/services/iamauth/mock_iamauth"
+	"sigs.k8s.io/cluster-api-provider-aws/v2/pkg/internal/testcert"
 	clusterv1 "sigs.k8s.io/cluster-api/api/v1beta1"
 )
 
 func TestOIDCReconcile(t *testing.T) {
+	testCertThumbprint := getTestcertTumbprint(t)
+
 	tests := []struct {
 		name    string
 		expect  func(m *mock_iamauth.MockIAMAPIMockRecorder, url string)
@@ -64,7 +72,7 @@ func TestOIDCReconcile(t *testing.T) {
 				}, nil)
 				m.CreateOpenIDConnectProvider(&iam.CreateOpenIDConnectProviderInput{
 					ClientIDList:   aws.StringSlice([]string{"sts.amazonaws.com"}),
-					ThumbprintList: aws.StringSlice([]string{"15dbd260c7465ecca6de2c0b2181187f66ee0d1a"}),
+					ThumbprintList: aws.StringSlice([]string{testCertThumbprint}),
 					Url:            &url,
 				}).Return(&iam.CreateOpenIDConnectProviderOutput{
 					OpenIDConnectProviderArn: aws.String("arn::oidc"),
@@ -102,7 +110,7 @@ func TestOIDCReconcile(t *testing.T) {
 					OpenIDConnectProviderArn: aws.String("arn::oidc"),
 				}).Return(&iam.GetOpenIDConnectProviderOutput{
 					ClientIDList:   aws.StringSlice([]string{"sts.amazonaws.com"}),
-					ThumbprintList: aws.StringSlice([]string{"15dbd260c7465ecca6de2c0b2181187f66ee0d1a"}),
+					ThumbprintList: aws.StringSlice([]string{testCertThumbprint}),
 					Url:            &url,
 				}, nil)
 				m.TagOpenIDConnectProvider(&iam.TagOpenIDConnectProviderInput{
@@ -125,11 +133,7 @@ func TestOIDCReconcile(t *testing.T) {
 			_ = ekscontrolplanev1.AddToScheme(scheme)
 			_ = corev1.AddToScheme(scheme)
 
-			ts := httptest.NewTLSServer(http.HandlerFunc(func(rw http.ResponseWriter, req *http.Request) {
-				// Send response to be tested
-				rw.WriteHeader(http.StatusOK)
-				rw.Write([]byte(`OK`))
-			}))
+			ts := createTestServer(g)
 			defer ts.Close()
 
 			controlPlane := &ekscontrolplanev1.AWSManagedControlPlane{
@@ -178,6 +182,41 @@ func TestOIDCReconcile(t *testing.T) {
 	}
 }
 
+func getTestcertTumbprint(t *testing.T) string {
+	t.Helper()
+	g := NewWithT(t)
+
+	block, _ := pem.Decode(testcert.LocalhostCert)
+	g.Expect(block).ToNot(BeNil(), "failed to parse certificate PEM")
+
+	x509Cert, err := x509.ParseCertificate(block.Bytes)
+	g.Expect(err).NotTo(HaveOccurred())
+
+	hash := sha1.Sum(x509Cert.Raw) //nolint:gosec
+	return hex.EncodeToString(hash[:])
+}
+
+func createTestServer(g *GomegaWithT) *httptest.Server {
+	// Create a certificate and private key
+	cert, err := tls.X509KeyPair(testcert.LocalhostCert, testcert.LocalhostKey)
+	g.Expect(err).NotTo(HaveOccurred())
+
+	// Create custom TLS config
+	tlsConfig := &tls.Config{ //nolint:gosec
+		Certificates: []tls.Certificate{cert},
+	}
+
+	// Create test server with custom TLS config
+	server := httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		w.WriteHeader(http.StatusOK)
+		w.Write([]byte(`OK`))
+	}))
+	server.TLS = tlsConfig
+	server.StartTLS()
+
+	return server
+}
+
 var kubeConfig = []byte(`apiVersion: v1
 clusters:
 - cluster:
diff --git a/pkg/internal/testcert/testcert.go b/pkg/internal/testcert/testcert.go
new file mode 100644
index 000000000..168309145
--- /dev/null
+++ b/pkg/internal/testcert/testcert.go
@@ -0,0 +1,62 @@
+// Package testcert contains a test-only localhost certificate.
+// NOTE: this is based on the code from the Go stdlib.
+package testcert
+
+import "strings"
+
+// LocalhostCert is a PEM-encoded TLS cert with SAN IPs
+// "127.0.0.1" and "[::1]", expiring at Jan 29 16:00:00 2084 GMT.
+// generated from src/crypto/tls:
+// go run generate_cert.go  --rsa-bits 2048 --host 127.0.0.1,::1,example.com --ca --start-date "Jan 1 00:00:00 1970" --duration=1000000h.
+var LocalhostCert = []byte(`-----BEGIN CERTIFICATE-----
+MIIDOTCCAiGgAwIBAgIQSRJrEpBGFc7tNb1fb5pKFzANBgkqhkiG9w0BAQsFADAS
+MRAwDgYDVQQKEwdBY21lIENvMCAXDTcwMDEwMTAwMDAwMFoYDzIwODQwMTI5MTYw
+MDAwWjASMRAwDgYDVQQKEwdBY21lIENvMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8A
+MIIBCgKCAQEA6Gba5tHV1dAKouAaXO3/ebDUU4rvwCUg/CNaJ2PT5xLD4N1Vcb8r
+bFSW2HXKq+MPfVdwIKR/1DczEoAGf/JWQTW7EgzlXrCd3rlajEX2D73faWJekD0U
+aUgz5vtrTXZ90BQL7WvRICd7FlEZ6FPOcPlumiyNmzUqtwGhO+9ad1W5BqJaRI6P
+YfouNkwR6Na4TzSj5BrqUfP0FwDizKSJ0XXmh8g8G9mtwxOSN3Ru1QFc61Xyeluk
+POGKBV/q6RBNklTNe0gI8usUMlYyoC7ytppNMW7X2vodAelSu25jgx2anj9fDVZu
+h7AXF5+4nJS4AAt0n1lNY7nGSsdZas8PbQIDAQABo4GIMIGFMA4GA1UdDwEB/wQE
+AwICpDATBgNVHSUEDDAKBggrBgEFBQcDATAPBgNVHRMBAf8EBTADAQH/MB0GA1Ud
+DgQWBBStsdjh3/JCXXYlQryOrL4Sh7BW5TAuBgNVHREEJzAlggtleGFtcGxlLmNv
+bYcEfwAAAYcQAAAAAAAAAAAAAAAAAAAAATANBgkqhkiG9w0BAQsFAAOCAQEAxWGI
+5NhpF3nwwy/4yB4i/CwwSpLrWUa70NyhvprUBC50PxiXav1TeDzwzLx/o5HyNwsv
+cxv3HdkLW59i/0SlJSrNnWdfZ19oTcS+6PtLoVyISgtyN6DpkKpdG1cOkW3Cy2P2
++tK/tKHRP1Y/Ra0RiDpOAmqn0gCOFGz8+lqDIor/T7MTpibL3IxqWfPrvfVRHL3B
+grw/ZQTTIVjjh4JBSW3WyWgNo/ikC1lrVxzl4iPUGptxT36Cr7Zk2Bsg0XqwbOvK
+5d+NTDREkSnUbie4GeutujmX3Dsx88UiV6UY/4lHJa6I5leHUNOHahRbpbWeOfs/
+WkBKOclmOV2xlTVuPw==
+-----END CERTIFICATE-----`)
+
+// LocalhostKey is the private key for LocalhostCert.
+var LocalhostKey = []byte(testingKey(`-----BEGIN RSA TESTING KEY-----
+MIIEvAIBADANBgkqhkiG9w0BAQEFAASCBKYwggSiAgEAAoIBAQDoZtrm0dXV0Aqi
+4Bpc7f95sNRTiu/AJSD8I1onY9PnEsPg3VVxvytsVJbYdcqr4w99V3AgpH/UNzMS
+gAZ/8lZBNbsSDOVesJ3euVqMRfYPvd9pYl6QPRRpSDPm+2tNdn3QFAvta9EgJ3sW
+URnoU85w+W6aLI2bNSq3AaE771p3VbkGolpEjo9h+i42TBHo1rhPNKPkGupR8/QX
+AOLMpInRdeaHyDwb2a3DE5I3dG7VAVzrVfJ6W6Q84YoFX+rpEE2SVM17SAjy6xQy
+VjKgLvK2mk0xbtfa+h0B6VK7bmODHZqeP18NVm6HsBcXn7iclLgAC3SfWU1jucZK
+x1lqzw9tAgMBAAECggEABWzxS1Y2wckblnXY57Z+sl6YdmLV+gxj2r8Qib7g4ZIk
+lIlWR1OJNfw7kU4eryib4fc6nOh6O4AWZyYqAK6tqNQSS/eVG0LQTLTTEldHyVJL
+dvBe+MsUQOj4nTndZW+QvFzbcm2D8lY5n2nBSxU5ypVoKZ1EqQzytFcLZpTN7d89
+EPj0qDyrV4NZlWAwL1AygCwnlwhMQjXEalVF1ylXwU3QzyZ/6MgvF6d3SSUlh+sq
+XefuyigXw484cQQgbzopv6niMOmGP3of+yV4JQqUSb3IDmmT68XjGd2Dkxl4iPki
+6ZwXf3CCi+c+i/zVEcufgZ3SLf8D99kUGE7v7fZ6AQKBgQD1ZX3RAla9hIhxCf+O
+3D+I1j2LMrdjAh0ZKKqwMR4JnHX3mjQI6LwqIctPWTU8wYFECSh9klEclSdCa64s
+uI/GNpcqPXejd0cAAdqHEEeG5sHMDt0oFSurL4lyud0GtZvwlzLuwEweuDtvT9cJ
+Wfvl86uyO36IW8JdvUprYDctrQKBgQDycZ697qutBieZlGkHpnYWUAeImVA878sJ
+w44NuXHvMxBPz+lbJGAg8Cn8fcxNAPqHIraK+kx3po8cZGQywKHUWsxi23ozHoxo
++bGqeQb9U661TnfdDspIXia+xilZt3mm5BPzOUuRqlh4Y9SOBpSWRmEhyw76w4ZP
+OPxjWYAgwQKBgA/FehSYxeJgRjSdo+MWnK66tjHgDJE8bYpUZsP0JC4R9DL5oiaA
+brd2fI6Y+SbyeNBallObt8LSgzdtnEAbjIH8uDJqyOmknNePRvAvR6mP4xyuR+Bv
+m+Lgp0DMWTw5J9CKpydZDItc49T/mJ5tPhdFVd+am0NAQnmr1MCZ6nHxAoGABS3Y
+LkaC9FdFUUqSU8+Chkd/YbOkuyiENdkvl6t2e52jo5DVc1T7mLiIrRQi4SI8N9bN
+/3oJWCT+uaSLX2ouCtNFunblzWHBrhxnZzTeqVq4SLc8aESAnbslKL4i8/+vYZlN
+s8xtiNcSvL+lMsOBORSXzpj/4Ot8WwTkn1qyGgECgYBKNTypzAHeLE6yVadFp3nQ
+Ckq9yzvP/ib05rvgbvrne00YeOxqJ9gtTrzgh7koqJyX1L4NwdkEza4ilDWpucn0
+xiUZS4SoaJq6ZvcBYS62Yr1t8n09iG47YL8ibgtmH3L+svaotvpVxVK+d7BLevA/
+ZboOWVe3icTy64BT3OQhmg==
+-----END RSA TESTING KEY-----`))
+
+func testingKey(s string) string { return strings.ReplaceAll(s, "TESTING KEY", "PRIVATE KEY") }