Repository: skupperproject/skupper
Pull request #2092 (2025-04-29 17:08:45)

update to go 1.24

ajssmith: Yes, working through those issues now. Leaning towards cleaning them up and getting to 1.24. I will push an update soon...

Patch:
diff --git a/.circleci/config.yml b/.circleci/config.yml
index 83e3870d..2ccc5741 100644
--- a/.circleci/config.yml
+++ b/.circleci/config.yml
@@ -8,7 +8,7 @@ orbs:
 executors:
   go_cimg:
     docker:
-      - image: cimg/go:1.22
+      - image: cimg/go:1.24
 
   local_cluster_test_executor:
     machine:
diff --git a/Dockerfile.cli b/Dockerfile.cli
index a3b44668..3495b9ec 100644
--- a/Dockerfile.cli
+++ b/Dockerfile.cli
@@ -1,4 +1,4 @@
-FROM --platform=$BUILDPLATFORM golang:1.22 AS builder
+FROM --platform=$BUILDPLATFORM golang:1.24 AS builder
 
 ARG TARGETARCH
 
diff --git a/Dockerfile.controller b/Dockerfile.controller
index e5929f19..bf39a398 100644
--- a/Dockerfile.controller
+++ b/Dockerfile.controller
@@ -1,4 +1,4 @@
-FROM --platform=$BUILDPLATFORM golang:1.22 AS builder
+FROM --platform=$BUILDPLATFORM golang:1.24 AS builder
 
 ARG TARGETARCH
 
diff --git a/Dockerfile.kube-adaptor b/Dockerfile.kube-adaptor
index 9b5a2602..a95a00cd 100644
--- a/Dockerfile.kube-adaptor
+++ b/Dockerfile.kube-adaptor
@@ -1,4 +1,4 @@
-FROM --platform=$BUILDPLATFORM golang:1.22 AS builder
+FROM --platform=$BUILDPLATFORM golang:1.24 AS builder
 
 ARG TARGETARCH
 
diff --git a/Dockerfile.network-observer b/Dockerfile.network-observer
index a4e1331f..a6f1f1a6 100644
--- a/Dockerfile.network-observer
+++ b/Dockerfile.network-observer
@@ -1,4 +1,4 @@
-FROM --platform=$BUILDPLATFORM golang:1.22 AS builder
+FROM --platform=$BUILDPLATFORM golang:1.24 AS builder
 
 ARG TARGETARCH
 
diff --git a/doc/tls/.gitignore b/doc/tls/.gitignore
new file mode 100644
index 00000000..f46a7bcc
--- /dev/null
+++ b/doc/tls/.gitignore
@@ -0,0 +1,3 @@
+*.crt
+*.key
+*.csr
diff --git a/doc/tls/README.md b/doc/tls/README.md
new file mode 100644
index 00000000..dca3a384
--- /dev/null
+++ b/doc/tls/README.md
@@ -0,0 +1,317 @@
+# TLS
+
+Skupper uses TLS certificates to authenticate and secure communications between
+Skupper routers in a network through channels called Links. The Skupper
+controller for Kubernetes will, by default, take care of issuing the self
+signed TLS credentials used by Links. This document describes the requirements
+for these TLS credentials, the default scheme used by the Skupper Kubernetes
+controller, and how other certificate infrastructure could be included.
+
+## TLS Credentials Requirements
+
+Skupper uses mutual TLS (mTLS) to establish Links between Sites. The Site that
+initiates the Link, the linking site, acts as the TLS client. The Site with a
+RouterAccess, the accepting site, acts as the TLS server.
+
+The accepting site must present a valid server certificate, and will require
+incoming connections to present a valid client certificate, which it validates
+against its trusted CAs. The linking site will initiate TLS connections,
+validating the server certificate against its own trusted CAs and presenting
+its own client certificate to the server. Any intermediate load balancers
+between Sites must use TLS passthrough. Terminating TLS will prevent the
+routers on either site from authenticating one another.
+
+### Link TLS Credentials (TLS client)
+
+A Link has its own set of TLS Credentials that includes a certificate and a
+database of trusted Certificate Authorities (CAs) that it will use to
+authenticate the server's certificate. The Link certificate has few specific
+requirements. It must be valid and signed by a CA the peer RouterAccess trusts,
+and it must have appropriate key usage attributes for client authentication.
+
+### RouterAccess TLS Credentials (TLS Server)
+
+Every Skupper Link is made from a Site, the linking Site, to the endpoints from
+the RouterAccess on a remote Site, the accepting Site. Each RouterAccess has
+its own set of TLS Credentials that includes the serving certificate for
+authenticating itself with peer routers, and a bundle of trusted Certificate
+Authorities (CAs) it uses to authenticate client certificates.
+
+The RouterAccess certificate has the typical requirements for a TLS web server:
+usage for digital signature, key encipherment and server auth. The certificate
+also must be valid for the host(s) in the Link/RouterAccess endpoints in order
+for peers to validate the connection.
+
+> ⚠️ Known Issue: The Skupper router ignores Subject Alternative Name IP entries
+> when doing hostname validation. The skupper controller works around this by
+> adding IPs as DNS entries. Not all PKI tools make this easy to configure. For
+> example, inspecting a certificate with `openssl x509 -ext subjectAltName`.
+>
+> X509v3 Subject Alternative Name:
+>   DNS:skupper-router, DNS:172.18.255.193, IP Address:172.18.255.193
+
+### Certificate Secret Layout
+
+Skupper expects TLS Credentials in standard Kubernetes kubernetes.io/tls Secret
+format with the following fields.
+
+- **tls.crt** PEM encoded X.509 certificate - the public certificate
+- **tls.key** PKCS#1 private key - the private key for the certificate
+- **ca.crt** PEM encoded X.509 certificate(s), trusted Certificate Authorities
+  "database". Multiple certificates can be concatenated in this field.
+
+## Default Controller-managed TLS in Kubernetes
+
+The Skupper controller for Kubernetes will automate the creation of the TLS
+Credentials for a Site's Links and RouterAccess. In this model, there is no
+central authority of trust for a network. Instead, each Site acts as its own
+root of trust, issuing and validating credentials independently using a
+site-scoped CA.
+
+### Issuing Site Link Access Credentials
+
+When a Site with link access enabled is initialized:
+- Skupper automatically creates a RouterAccess resource for the Site named
+  `skupper-router`.
+- Skupper issues a self-signed CA named `skupper-site-ca`, valid for 5 years.
+- Skupper issues a `skupper-site-server` certificate, valid for 5 years, signed
+  by `skupper-site-ca`.
+- The `skupper-site-ca` certificate is embedded in the `ca.crt` field of
+  `skupper-site-server`.
+- The RouterAccess is configured with the `skupper-site-server` TLS
+  credentials. Only clients presenting certificates signed by `skupper-site-ca`
+  will be authorized to connect.
+
+### Issuing Link Credentials
+
+A Link can be created in several ways, either manually with `skupper link
+generate` or when redeeming an AccessToken. Regardless, the issuance of TLS
+credentials is done by the accepting Site.
+
+- The Skupper controller issues a new client TLS certificate signed by the
+  accepting Site's `skupper-site-ca` CA, and embeds the CA public key into the
+  `ca.crt` field of the client certificate.
+- The new client certificate is transported to the linking Site, either
+  manually by the user or over https through the AccessToken redemption
+  endpoint. It is saved as a Secret, often with a random name based on the
+  accepting site name.
+- This Secret is then referenced in the Link’s spec.tlsCredentials field.
+- The linking Site's routers are configured, and secure connections to
+  accepting Site routers using these new credentials are made.
+
+## Manual TLS Certificate Management
+
+Understanding the requirements for TLS Credentials for connecting Sites and the
+default behaviour of the Skupper controller for Kubernetes, it is possible to
+exercise finer control over the certificates issued.
+
+### Using user-provided CAs
+
+As a rule, if a secret that the Skupper controller requires already exists, or
+that secret is overwritten by a user, Skupper will stop trying to manage that
+secret. This means that it is possible to provide any of the credentials
+manually. One pattern that can be useful is to provide Sites with an alternate
+CA in `skupper-site-ca`.
+
+### Manually Managing Link and RouterAccess tlsCredentials
+
+Full manual control of TLS certificates can be accomplished by manually
+managing RouterAccess and Links.
+
+This example sets up the following:
+* A toy PKI using the `openssl` tool (written using `OpenSSL 3.2.4`)
+* A single network-scoped self-signed CA for simplicity - contrasts with the default site-scoped CA.
+* Two Kubernetes sites in different clusters. One "public" Site that will use a
+  LoadBalancer Service for ingress, and one "private" Site without ingress.
+
+#### Issue a network CA
+
+Use the openssl configuration file in this directory to set up a self-signed CA.
+```
+openssl req -x509 -new \
+        -config network-ca.conf -newkey rsa \
+        -out ca.crt -keyout ca.key
+```
+
+This should produce two files `ca.crt` and `ca.key`.
+
+#### Set up the Skupper Sites
+
+Using the kube context for the public site: `kubectl apply -f public.yaml`
+
+This manifest has a Site and RouterAccess configured for the public site. The
+RouterAccess points to `tlsCredentials: public-server-tls`. This will be the
+name of the Secret Skupper will use.
+
+Using the kube context for the private site: `kubectl apply -f private.yaml`
+
+#### Manually Issuing Certificates
+
+In order to issue a TLS certificate for the public site, we first need the
+address(es) that the RouterAccess will listen on. This is required for peers to
+validate the host.
+
+Get the RouterAccess Status using the kube context for the public site:
+```
+kubectl get routeraccesses.skupper.io skupper \
+    -ojsonpath='{range .status.endpoints[*]}{.host}{"\n"}{end}'
+```
+
+Assuming that a LoadBalancer service was created, the RouterAccess status
+should be populated with a list of endpoints. This will be important later, but
+for now we only need the hosts. In this example there was one unique address,
+`172.18.255.193`.
+
+Now we can issue a key pair for the public site and put it in the `public-server-tls` Secret.
+
+> ⚠️ Do not forget:  replace  COMMON_NAME with your RouterAccess host.
+
+```
+COMMON_NAME=172.18.255.193
+# Generate a private key for site-public
+openssl genrsa -out site-public.key 4096
+# Create a signing request, setting the CN to the address
+# from the RouterAccess endpoint
+openssl req -new \
+    -config network-ca.conf -section site \
+    -subj "/CN=${COMMON_NAME:-skupper-router}/O=network:site" \
+    -key site-public.key -out site-public.csr
+# Issue the public site's certificate
+openssl x509 -req -days 100 \
+    -copy_extensions copyall \
+    -CA ca.crt -CAkey ca.key \
+    -in site-public.csr -out site-public.crt
+```
+
+This should produce the following files:
+
+`site-public.key`: The private key for the public site
+`site-public.crt`: The public key of the public site
+
+Add these new credentials to the public site.
+
+```
+kubectl create secret generic public-server-tls \
+    --from-file=ca.crt=ca.crt \
+    --from-file=tls.crt=site-public.crt \
+    --from-file=tls.key=site-public.key
+```
+Now we can issue a key pair for the private site to use for client authentication with the public site.
+
+```
+# Generate a private key
+openssl genrsa -out site-private.key 4096
+# Create a signing request for the private key
+openssl req -new \
+    -config network-ca.conf -section site \
+    -key site-private.key -out site-private.csr
+# Issue the private site's certificate
+openssl x509 -req -days 100 \
+    -copy_extensions copyall \
+    -CA ca.crt -CAkey ca.key \
+    -in site-private.csr -out site-private.crt
+```
+
+This should produce the following files:
+
+`site-private.key`: The private key for the private site
+`site-private.crt`: The public key of the private site
+
+Add these new credentials to the private site.
+```
+kubectl create secret generic public-link-tls \
+    --from-file=ca.crt=ca.crt \
+    --from-file=tls.crt=site-private.crt \
+    --from-file=tls.key=site-private.key
+```
+
+#### Link Sites
+
+At this point the public site should have a working RouterAccess and the
+private site should have TLS credentials (client side) that will allow it to be authenticated by the public site (server side) and vice-versa.
+The private site needs a Link resource to complete the connection.
+
+The Link document needs to be populated with `tlsCredentials: public-link-tls`
+to match the name of the secret in the private site to use. It also requires a
+`endpoints` block, identical to the one in the public site's RouterAccess
+status section. The skupper CLI can generate this document for you, using the
+public kube context.
+
+```
+skupper link generate \
+    --tls-credentials public-link-tls \
+    --generate-credential=false | tee link.yaml
+```
+
+This will output a Link record document with the endpoints filled to connect to
+the public site.
+
+Using the kube context for the private site: `kubectl apply -f link.yaml`
+
+This will apply the link to the private site. The sites should connect.
+
+## TLS Credential Rotation
+
+**TODO** Support for rotating TLS Credentials is currently under development
+and is not yet fully implemented.
+
+When considering rotating the TLS credentials used by Skupper, it is important
+to understand the trust model used for linking Skupper Sites. The default
+[trust model](#default-controller-managed-tls-in-kubernetes) used by Skupper is
+distributed: each Site having its own trust root. Because of this, rotating
+client certificates and Site CAs is an especially complicated exercise.
+
+## Troubleshooting TLS Issues
+
+TLS errors logged by Skupper routers are relatively common, and do not always
+indicate a problem on their own. Because Skupper routers use TLS connections
+for everything, ANY connectivity issue will surface as a TLS error. For
+example, any router in the network getting rescheduled or stopped will likely
+manifest a TLS error somewhere.
+
+When there is a Link in the network that appears broken, begin by verifying
+connectivity before looking at TLS specific issues.
+
+- Check that the destination Site Status for problems.
+  - Is the Site Ready? Are the skupper-router deployment(s) running?
+  - Does the Site Status Endpoints match what is configured in the Link?
+  - Is the ingress for the router ready? This depends on your controller and
+    site configuration. Could be a LoadBalancer or NodePort Service, OpenShift
+    Route, Gateway API TLSRoute, etc.
+- Check for connectivity problems to the Link endpoints.
+  - Find the host:port combinations from the Link's endpoints
+  - Use a TCP or TLS client to test connectivity, ideally from the same
+    namespace as the Link's Site to catch any network policy issues. Any of the
+    following common clients will work.
+    - TCP: `echo "hello" | nc <host> <port>` Sends nonsense to the router. Expect a
+      router to respond with an AMQPS error. Same for `telnet`.
+    - TLS: `curl --insecure https://<host>:<port>` attempts to open a TLS connection
+      (for an http request). Expect a router to refuse the connection due to
+      missing client certificates. curl should print an SSL error.
+    - TLS: `openssl s_client -showcerts -connect <host>:<port>` Displays diagnostics
+      about an attempt to open a TLS connection. Expect a router to refuse the
+      connection due to missing client certificates.
+
+When connectivity can be verified, it can be helpful to look into validating
+TLS certificates on either side of the connection if the Link is still not
+operational.
+
+A quick test can be done using the linking site's router deployment. The
+following command uses the openssl tool to attempt to open a TLS connection to
+a peer router, and displays detailed diagnostics. Substitute the name of the
+TLS Credentials secret used by the link, the Link endpoint host, and the
+relevant port (likely 55671.)
+```
+TLS_SECRET_NAME=public-link-tls
+LINK_ENDPOINT_HOST=172.18.255.193
+PORT=55671
+PROFILE_DIR="/etc/skupper-router-certs/${TLS_SECRET_NAME}-profile"
+kubectl exec -it deployments/skupper-router \
+    -c router -- \
+        openssl s_client \
+        -CAfile "$PROFILE_DIR/ca.crt" \
+        -cert "$PROFILE_DIR/tls.crt" \
+        -key "$PROFILE_DIR/tls.key" \
+        -connect "$LINK_ENDPOINT_HOST:$PORT"  < /dev/null
+```
+
diff --git a/doc/tls/network-ca.conf b/doc/tls/network-ca.conf
new file mode 100644
index 00000000..11b66c11
--- /dev/null
+++ b/doc/tls/network-ca.conf
@@ -0,0 +1,30 @@
+[req]
+prompt             = no
+encrypt_key        = no
+default_bits       = 4096
+default_md         = sha512
+distinguished_name = ca_dn
+x509_extensions    = ca_ext
+
+[ca_ext]
+authorityKeyIdentifier = keyid:always,issuer
+basicConstraints       = critical, CA:true
+keyUsage               = critical, digitalSignature, keyCertSign
+
+[ca_dn]
+O  = network
+CN = Skupper Network CA
+
+[site]
+distinguished_name = site_dn
+prompt             = no
+req_extensions     = site_ext
+
+[site_dn]
+CN = skupper-site
+O  = network:site
+
+[site_ext]
+basicConstraints       = CA:FALSE
+keyUsage               = critical, nonRepudiation, digitalSignature, keyEncipherment
+extendedKeyUsage       = clientAuth, serverAuth
diff --git a/doc/tls/private.yaml b/doc/tls/private.yaml
new file mode 100644
index 00000000..be6bacbe
--- /dev/null
+++ b/doc/tls/private.yaml
@@ -0,0 +1,7 @@
+apiVersion: skupper.io/v2alpha1
+kind: Site
+metadata:
+  name: private
+spec:
+  defaultIssuer: none
+
diff --git a/doc/tls/public.yaml b/doc/tls/public.yaml
new file mode 100644
index 00000000..0eac44ff
--- /dev/null
+++ b/doc/tls/public.yaml
@@ -0,0 +1,20 @@
+apiVersion: skupper.io/v2alpha1
+kind: Site
+metadata:
+  name: public
+spec:
+  defaultIssuer: none
+---
+apiVersion: skupper.io/v2alpha1
+kind: RouterAccess
+metadata:
+  name: skupper
+spec:
+  issuer: none
+  tlsCredentials: public-server-tls
+  accessType: loadbalancer
+  roles:
+    - port: 55671
+      name: inter-router
+    - port: 45671
+      name: edge
diff --git a/go.mod b/go.mod
index 7160dafe..3e085529 100644
--- a/go.mod
+++ b/go.mod
@@ -1,19 +1,16 @@
 module github.com/skupperproject/skupper
 
-go 1.22.0
+go 1.24.0
 
-toolchain go1.22.8
+toolchain go1.24.2
 
 require (
 	github.com/Azure/go-amqp v1.0.5
 	github.com/briandowns/spinner v1.23.0
 	github.com/cenkalti/backoff/v4 v4.3.0
 	github.com/fsnotify/fsnotify v1.7.0
-	github.com/go-openapi/errors v0.20.3
 	github.com/go-openapi/runtime v0.24.1
 	github.com/go-openapi/strfmt v0.21.3
-	github.com/go-openapi/swag v0.22.8
-	github.com/go-openapi/validate v0.22.0
 	github.com/google/go-cmp v0.6.0
 	github.com/google/uuid v1.6.0
 	github.com/gorilla/handlers v1.5.2
@@ -61,10 +58,13 @@ require (
 	github.com/getkin/kin-openapi v0.124.0 // indirect
 	github.com/go-logr/logr v1.4.2 // indirect
 	github.com/go-openapi/analysis v0.21.2 // indirect
+	github.com/go-openapi/errors v0.20.3 // indirect
 	github.com/go-openapi/jsonpointer v0.20.2 // indirect
 	github.com/go-openapi/jsonreference v0.20.2 // indirect
 	github.com/go-openapi/loads v0.21.1 // indirect
 	github.com/go-openapi/spec v0.20.4 // indirect
+	github.com/go-openapi/swag v0.22.8 // indirect
+	github.com/go-openapi/validate v0.22.0 // indirect
 	github.com/gogo/protobuf v1.3.2 // indirect
 	github.com/golang/protobuf v1.5.4 // indirect
 	github.com/google/gnostic-models v0.6.8 // indirect
diff --git a/internal/cmd/skupper/token/kube/token_issue.go b/internal/cmd/skupper/token/kube/token_issue.go
index 9a115593..f52ed89c 100644
--- a/internal/cmd/skupper/token/kube/token_issue.go
+++ b/internal/cmd/skupper/token/kube/token_issue.go
@@ -179,12 +179,12 @@ func (cmd *CmdTokenIssue) WaitUntil() error {
 
 			encodedResource, err := utils.Encode("yaml", accessToken)
 			if err != nil {
-				return fmt.Errorf("Could not write out generated token: " + err.Error())
+				return fmt.Errorf("could not write out generated token: %s", err.Error())
 			}
 
 			err = os.WriteFile(cmd.fileName, []byte(encodedResource), 0644)
 			if err != nil {
-				return fmt.Errorf("Could not write to file " + cmd.fileName + ": " + err.Error())
+				return fmt.Errorf("could not write to file %s:%s", cmd.fileName, err.Error())
 			}
 
 			return nil
@@ -194,7 +194,7 @@ func (cmd *CmdTokenIssue) WaitUntil() error {
 	})
 
 	if err != nil {
-		return fmt.Errorf("Grant %q not ready yet, check the status for more information\n", cmd.grantName)
+		return fmt.Errorf("grant %q not ready yet, check the status for more information", cmd.grantName)
 	}
 
 	fmt.Printf("\nGrant %q is ready\n", cmd.grantName)
diff --git a/internal/kube/client/fake/fake_client.go b/internal/kube/client/fake/fake_client.go
index 66e3f9ae..ed3462bd 100644
--- a/internal/kube/client/fake/fake_client.go
+++ b/internal/kube/client/fake/fake_client.go
@@ -45,7 +45,7 @@ func NewFakeClient(namespace string, k8sObjects []runtime.Object, skupperObjects
 	// Note: brute force error return for any client access, we could make it more granular if needed
 	if fakeSkupperError != "" {
 		c.Skupper.SkupperV2alpha1().(*fakeskupperv2alpha1.FakeSkupperV2alpha1).PrependReactor("*", "*", func(action testing.Action) (handled bool, ret runtime.Object, err error) {
-			return true, nil, fmt.Errorf(fakeSkupperError)
+			return true, nil, fmt.Errorf("%s", fakeSkupperError)
 		})
 	}
 	scheme := runtime.NewScheme()
@@ -65,9 +65,7 @@ func NewFakeClient(namespace string, k8sObjects []runtime.Object, skupperObjects
 	}
 	c.Discovery = c.Skupper.Discovery()
 	if fakeDiscoveryClient, ok := c.Discovery.(*discoveryfake.FakeDiscovery); ok {
-		for _, resources := range fakedApiResources() {
-			fakeDiscoveryClient.Resources = append(fakeDiscoveryClient.Resources, resources)
-		}
+		fakeDiscoveryClient.Resources = append(fakeDiscoveryClient.Resources, fakedApiResources()...)
 	}
 	c.Route = routefake.NewSimpleClientset(routes...)
 
diff --git a/internal/kube/grants/grants.go b/internal/kube/grants/grants.go
index b268735a..dad94cc9 100644
--- a/internal/kube/grants/grants.go
+++ b/internal/kube/grants/grants.go
@@ -213,7 +213,7 @@ func (g *Grants) checkGrant(key string, grant *skupperv2alpha1.AccessGrant) erro
 	var err error
 
 	if len(status) != 0 {
-		err = fmt.Errorf(strings.Join(status, ", "))
+		err = fmt.Errorf("%s", strings.Join(status, ", "))
 	}
 	if grant.SetProcessed(err) {
 		changed = true
diff --git a/internal/kube/site/binding_status.go b/internal/kube/site/binding_status.go
index 43e2cd9b..2244f2bf 100644
--- a/internal/kube/site/binding_status.go
+++ b/internal/kube/site/binding_status.go
@@ -86,7 +86,7 @@ func (s *BindingStatus) updateMatchingListenerCountForAttachedConnector(connecto
 
 func (s *BindingStatus) error() error {
 	if len(s.errors) > 0 {
-		return fmt.Errorf(strings.Join(s.errors, ", "))
+		return fmt.Errorf("%s", strings.Join(s.errors, ", "))
 	}
 	return nil
 }
diff --git a/internal/kube/site/site.go b/internal/kube/site/site.go
index acb13d6c..5d6918dd 100644
--- a/internal/kube/site/site.go
+++ b/internal/kube/site/site.go
@@ -1258,7 +1258,7 @@ func (s *Site) CheckRouterAccess(name string, la *skupperv2alpha1.RouterAccess)
 		}
 		var err error
 		if len(errors) > 0 {
-			err = fmt.Errorf(strings.Join(errors, ", "))
+			err = fmt.Errorf("%s", strings.Join(errors, ", "))
 		}
 		if la != nil && la.SetConfigured(err) {
 			s.updateRouterAccessStatus(la)
diff --git a/internal/nonkube/common/systemd.go b/internal/nonkube/common/systemd.go
index aa0f625f..1153fed7 100644
--- a/internal/nonkube/common/systemd.go
+++ b/internal/nonkube/common/systemd.go
@@ -95,7 +95,7 @@ func (s *systemdServiceInfo) Create() error {
 		if s.getUid() != 0 {
 			msg += " at user level"
 		}
-		return fmt.Errorf(msg)
+		return fmt.Errorf("%s", msg)
 	}
 	var logger = NewLogger()
 	logger.Debug("creating systemd service")
diff --git a/internal/qdr/amqp_mgmt.go b/internal/qdr/amqp_mgmt.go
index ee35f54b..577ba15b 100644
--- a/internal/qdr/amqp_mgmt.go
+++ b/internal/qdr/amqp_mgmt.go
@@ -603,7 +603,7 @@ func (a *Agent) BatchQuery(queries []Query) ([][]Record, error) {
 		}
 	}
 	if len(errors) > 0 {
-		return nil, fmt.Errorf(strings.Join(errors, ", "))
+		return nil, fmt.Errorf("%s", strings.Join(errors, ", "))
 	}
 	return batchResults, nil
 }
diff --git a/internal/utils/formatter/list.go b/internal/utils/formatter/list.go
index e961e7d0..6e023539 100644
--- a/internal/utils/formatter/list.go
+++ b/internal/utils/formatter/list.go
@@ -62,7 +62,7 @@ func printList(l *list, level int, last map[int]bool) {
 			fmt.Printf("╰─ ")
 		}
 	}
-	fmt.Printf(l.item)
+	fmt.Printf("%s", l.item)
 	if l.details != nil && len(l.details.details) > 0 {
 		itHasChildren := len(l.children) > 0
 		printDetails(l.details.details, level, last, itHasChildren)


Test patch:
diff --git a/Dockerfile.ci-test b/Dockerfile.ci-test
index 0adb05ba..521a327c 100644
--- a/Dockerfile.ci-test
+++ b/Dockerfile.ci-test
@@ -1,4 +1,4 @@
-FROM --platform=$BUILDPLATFORM golang:1.22 AS builder
+FROM --platform=$BUILDPLATFORM golang:1.24 AS builder
 
 ARG TARGETOS
 ARG TARGETARCH
diff --git a/internal/cmd/skupper/connector/kube/connector_create_test.go b/internal/cmd/skupper/connector/kube/connector_create_test.go
index b7e1175d..df674771 100644
--- a/internal/cmd/skupper/connector/kube/connector_create_test.go
+++ b/internal/cmd/skupper/connector/kube/connector_create_test.go
@@ -548,8 +548,18 @@ func TestCmdConnectorCreate_InputToOptions(t *testing.T) {
 
 	testTable := []test{
 		{
-			name:                   "test1",
-			flags:                  common.CommandConnectorCreateFlags{"backend", "", "app=backend", "secret", "tcp", true, "", 20 * time.Second, "ready"},
+			name: "test1",
+			flags: common.CommandConnectorCreateFlags{
+				RoutingKey:          "backend",
+				Host:                "",
+				Selector:            "app=backend",
+				TlsCredentials:      "secret",
+				ConnectorType:       "tcp",
+				IncludeNotReadyPods: true,
+				Workload:            "",
+				Timeout:             20 * time.Second,
+				Wait:                "ready",
+			},
 			expectedTlsCredentials: "secret",
 			expectedHost:           "",
 			expectedRoutingKey:     "backend",
@@ -559,8 +569,18 @@ func TestCmdConnectorCreate_InputToOptions(t *testing.T) {
 			expectedStatus:         "ready",
 		},
 		{
-			name:                   "test2",
-			flags:                  common.CommandConnectorCreateFlags{"backend", "backend", "", "secret", "tcp", true, "", 20 * time.Second, "configured"},
+			name: "test2",
+			flags: common.CommandConnectorCreateFlags{
+				RoutingKey:          "backend",
+				Host:                "backend",
+				Selector:            "",
+				TlsCredentials:      "secret",
+				ConnectorType:       "tcp",
+				IncludeNotReadyPods: true,
+				Workload:            "",
+				Timeout:             20 * time.Second,
+				Wait:                "configured",
+			},
 			expectedTlsCredentials: "secret",
 			expectedHost:           "backend",
 			expectedRoutingKey:     "backend",
@@ -570,8 +590,18 @@ func TestCmdConnectorCreate_InputToOptions(t *testing.T) {
 			expectedStatus:         "configured",
 		},
 		{
-			name:                   "test3",
-			flags:                  common.CommandConnectorCreateFlags{"", "", "", "secret", "tcp", false, "", 30 * time.Second, "none"},
+			name: "test3",
+			flags: common.CommandConnectorCreateFlags{
+				RoutingKey:          "",
+				Host:                "",
+				Selector:            "",
+				TlsCredentials:      "secret",
+				ConnectorType:       "tcp",
+				IncludeNotReadyPods: false,
+				Workload:            "",
+				Timeout:             30 * time.Second,
+				Wait:                "none",
+			},
 			expectedTlsCredentials: "secret",
 			expectedHost:           "",
 			expectedRoutingKey:     "test3",
diff --git a/internal/cmd/skupper/connector/kube/connector_generate_test.go b/internal/cmd/skupper/connector/kube/connector_generate_test.go
index f4462eff..cb0f452a 100644
--- a/internal/cmd/skupper/connector/kube/connector_generate_test.go
+++ b/internal/cmd/skupper/connector/kube/connector_generate_test.go
@@ -213,8 +213,17 @@ func TestCmdConnectorGenerate_InputToOptions(t *testing.T) {
 
 	testTable := []test{
 		{
-			name:                   "test1",
-			flags:                  common.CommandConnectorGenerateFlags{"backend", "", "app=backend", "secret", "tcp", true, "", "json"},
+			name: "test1",
+			flags: common.CommandConnectorGenerateFlags{
+				RoutingKey:          "backend",
+				Host:                "",
+				Selector:            "app=backend",
+				TlsCredentials:      "secret",
+				ConnectorType:       "tcp",
+				IncludeNotReadyPods: true,
+				Workload:            "",
+				Output:              "json",
+			},
 			expectedTlsCredentials: "secret",
 			expectedHost:           "",
 			expectedRoutingKey:     "backend",
@@ -223,8 +232,17 @@ func TestCmdConnectorGenerate_InputToOptions(t *testing.T) {
 			expectedSelector:       "app=backend",
 		},
 		{
-			name:                   "test2",
-			flags:                  common.CommandConnectorGenerateFlags{"backend", "backend", "", "secret", "tcp", true, "", "json"},
+			name: "test2",
+			flags: common.CommandConnectorGenerateFlags{
+				RoutingKey:          "backend",
+				Host:                "backend",
+				Selector:            "",
+				TlsCredentials:      "secret",
+				ConnectorType:       "tcp",
+				IncludeNotReadyPods: true,
+				Workload:            "",
+				Output:              "json",
+			},
 			expectedTlsCredentials: "secret",
 			expectedHost:           "backend",
 			expectedRoutingKey:     "backend",
@@ -233,8 +251,17 @@ func TestCmdConnectorGenerate_InputToOptions(t *testing.T) {
 			expectedSelector:       "",
 		},
 		{
-			name:                   "test3",
-			flags:                  common.CommandConnectorGenerateFlags{"", "", "", "secret", "tcp", false, "", "yaml"},
+			name: "test3",
+			flags: common.CommandConnectorGenerateFlags{
+				RoutingKey:          "",
+				Host:                "",
+				Selector:            "",
+				TlsCredentials:      "secret",
+				ConnectorType:       "tcp",
+				IncludeNotReadyPods: false,
+				Workload:            "",
+				Output:              "yaml",
+			},
 			expectedTlsCredentials: "secret",
 			expectedHost:           "",
 			expectedRoutingKey:     "test3",
diff --git a/internal/cmd/skupper/connector/nonkube/connector_create_test.go b/internal/cmd/skupper/connector/nonkube/connector_create_test.go
index 397ce22a..cadc93c5 100644
--- a/internal/cmd/skupper/connector/nonkube/connector_create_test.go
+++ b/internal/cmd/skupper/connector/nonkube/connector_create_test.go
@@ -156,8 +156,18 @@ func TestNonKubeCmdConnectorCreate_InputToOptions(t *testing.T) {
 
 	testTable := []test{
 		{
-			name:                   "test1",
-			flags:                  common.CommandConnectorCreateFlags{"backend", "", "", "secret", "tcp", false, "", 0, "none"},
+			name: "test1",
+			flags: common.CommandConnectorCreateFlags{
+				RoutingKey:          "backend",
+				Host:                "",
+				Selector:            "",
+				TlsCredentials:      "secret",
+				ConnectorType:       "tcp",
+				IncludeNotReadyPods: false,
+				Workload:            "",
+				Timeout:             0,
+				Wait:                "none",
+			},
 			expectedTlsCredentials: "secret",
 			expectedHost:           "",
 			expectedRoutingKey:     "backend",
@@ -165,9 +175,19 @@ func TestNonKubeCmdConnectorCreate_InputToOptions(t *testing.T) {
 			expectedNamespace:      "default",
 		},
 		{
-			name:                   "test2",
-			namespace:              "test",
-			flags:                  common.CommandConnectorCreateFlags{"backend", "1.2.3.4", "", "secret", "tcp", false, "", 0, "configured"},
+			name:      "test2",
+			namespace: "test",
+			flags: common.CommandConnectorCreateFlags{
+				RoutingKey:          "backend",
+				Host:                "1.2.3.4",
+				Selector:            "",
+				TlsCredentials:      "secret",
+				ConnectorType:       "tcp",
+				IncludeNotReadyPods: false,
+				Workload:            "",
+				Timeout:             0,
+				Wait:                "configured",
+			},
 			expectedTlsCredentials: "secret",
 			expectedHost:           "1.2.3.4",
 			expectedRoutingKey:     "backend",
@@ -175,9 +195,19 @@ func TestNonKubeCmdConnectorCreate_InputToOptions(t *testing.T) {
 			expectedNamespace:      "test",
 		},
 		{
-			name:                   "test3",
-			namespace:              "test",
-			flags:                  common.CommandConnectorCreateFlags{"", "", "", "secret", "tcp", false, "", 0, "ready"},
+			name:      "test3",
+			namespace: "test",
+			flags: common.CommandConnectorCreateFlags{
+				RoutingKey:          "",
+				Host:                "",
+				Selector:            "",
+				TlsCredentials:      "secret",
+				ConnectorType:       "tcp",
+				IncludeNotReadyPods: false,
+				Workload:            "",
+				Timeout:             0,
+				Wait:                "ready",
+			},
 			expectedTlsCredentials: "secret",
 			expectedHost:           "",
 			expectedRoutingKey:     "my-Connector",
diff --git a/internal/cmd/skupper/connector/nonkube/connector_generate_test.go b/internal/cmd/skupper/connector/nonkube/connector_generate_test.go
index 4aea3f5b..cbcc7153 100644
--- a/internal/cmd/skupper/connector/nonkube/connector_generate_test.go
+++ b/internal/cmd/skupper/connector/nonkube/connector_generate_test.go
@@ -162,8 +162,17 @@ func TestNonKubeCmdConnectorGenerate_InputToOptions(t *testing.T) {
 
 	testTable := []test{
 		{
-			name:                   "test1",
-			flags:                  common.CommandConnectorGenerateFlags{"backend", "", "", "secret", "tcp", false, "", "json"},
+			name: "test1",
+			flags: common.CommandConnectorGenerateFlags{
+				RoutingKey:          "backend",
+				Host:                "",
+				Selector:            "",
+				TlsCredentials:      "secret",
+				ConnectorType:       "tcp",
+				IncludeNotReadyPods: false,
+				Workload:            "",
+				Output:              "json",
+			},
 			expectedTlsCredentials: "secret",
 			expectedHost:           "",
 			expectedRoutingKey:     "backend",
@@ -172,9 +181,18 @@ func TestNonKubeCmdConnectorGenerate_InputToOptions(t *testing.T) {
 			expectedNamespace:      "default",
 		},
 		{
-			name:                  "test2",
-			namespace:             "test",
-			flags:                 common.CommandConnectorGenerateFlags{"backend", "1.2.3.4", "", "", "tcp", false, "", "yaml"},
+			name:      "test2",
+			namespace: "test",
+			flags: common.CommandConnectorGenerateFlags{
+				RoutingKey:          "backend",
+				Host:                "1.2.3.4",
+				Selector:            "",
+				TlsCredentials:      "",
+				ConnectorType:       "tcp",
+				IncludeNotReadyPods: false,
+				Workload:            "",
+				Output:              "yaml",
+			},
 			expectedHost:          "1.2.3.4",
 			expectedRoutingKey:    "backend",
 			expectedConnectorType: "tcp",
@@ -182,9 +200,18 @@ func TestNonKubeCmdConnectorGenerate_InputToOptions(t *testing.T) {
 			expectedNamespace:     "test",
 		},
 		{
-			name:                   "test3",
-			namespace:              "test",
-			flags:                  common.CommandConnectorGenerateFlags{"", "", "", "secret", "tcp", false, "", "yaml"},
+			name:      "test3",
+			namespace: "test",
+			flags: common.CommandConnectorGenerateFlags{
+				RoutingKey:          "",
+				Host:                "",
+				Selector:            "",
+				TlsCredentials:      "secret",
+				ConnectorType:       "tcp",
+				IncludeNotReadyPods: false,
+				Workload:            "",
+				Output:              "yaml",
+			},
 			expectedTlsCredentials: "secret",
 			expectedHost:           "",
 			expectedRoutingKey:     "my-Connector",
diff --git a/internal/cmd/skupper/link/kube/link_status_test.go b/internal/cmd/skupper/link/kube/link_status_test.go
index f5c7d920..6b62e485 100644
--- a/internal/cmd/skupper/link/kube/link_status_test.go
+++ b/internal/cmd/skupper/link/kube/link_status_test.go
@@ -92,7 +92,9 @@ func TestCmdLinkStatus_InputToOptions(t *testing.T) {
 		cmd, err := newCmdLinkStatusWithMocks("test", nil, nil, "")
 		assert.Assert(t, err)
 
-		cmd.Flags = &common.CommandLinkStatusFlags{"json"}
+		cmd.Flags = &common.CommandLinkStatusFlags{
+			Output: "json",
+		}
 
 		cmd.InputToOptions()
 
diff --git a/internal/cmd/skupper/listener/kube/listener_create_test.go b/internal/cmd/skupper/listener/kube/listener_create_test.go
index ac8a59f5..e1354391 100644
--- a/internal/cmd/skupper/listener/kube/listener_create_test.go
+++ b/internal/cmd/skupper/listener/kube/listener_create_test.go
@@ -215,8 +215,15 @@ func TestCmdListenerCreate_InputToOptions(t *testing.T) {
 
 	testTable := []test{
 		{
-			name:                   "test1",
-			flags:                  common.CommandListenerCreateFlags{"backend", "backend", "secret", "tcp", 20 * time.Second, "configured"},
+			name: "test1",
+			flags: common.CommandListenerCreateFlags{
+				RoutingKey:     "backend",
+				Host:           "backend",
+				TlsCredentials: "secret",
+				ListenerType:   "tcp",
+				Timeout:        20 * time.Second,
+				Wait:           "configured",
+			},
 			expectedTlsCredentials: "secret",
 			expectedHost:           "backend",
 			expectedRoutingKey:     "backend",
@@ -225,8 +232,15 @@ func TestCmdListenerCreate_InputToOptions(t *testing.T) {
 			expectedStatus:         "configured",
 		},
 		{
-			name:                   "test2",
-			flags:                  common.CommandListenerCreateFlags{"", "", "secret", "tcp", 30 * time.Second, "ready"},
+			name: "test2",
+			flags: common.CommandListenerCreateFlags{
+				RoutingKey:     "",
+				Host:           "",
+				TlsCredentials: "secret",
+				ListenerType:   "tcp",
+				Timeout:        30 * time.Second,
+				Wait:           "ready",
+			},
 			expectedTlsCredentials: "secret",
 			expectedHost:           "test2",
 			expectedRoutingKey:     "test2",
diff --git a/internal/cmd/skupper/listener/kube/listener_generate_test.go b/internal/cmd/skupper/listener/kube/listener_generate_test.go
index a0383701..1296f6b6 100644
--- a/internal/cmd/skupper/listener/kube/listener_generate_test.go
+++ b/internal/cmd/skupper/listener/kube/listener_generate_test.go
@@ -171,8 +171,14 @@ func TestCmdListenerGenerate_InputToOptions(t *testing.T) {
 
 	testTable := []test{
 		{
-			name:                   "test1",
-			flags:                  common.CommandListenerGenerateFlags{"backend", "backend", "secret", "tcp", "json"},
+			name: "test1",
+			flags: common.CommandListenerGenerateFlags{
+				RoutingKey:     "backend",
+				Host:           "backend",
+				TlsCredentials: "secret",
+				ListenerType:   "tcp",
+				Output:         "json",
+			},
 			expectedTlsCredentials: "secret",
 			expectedHost:           "backend",
 			expectedRoutingKey:     "backend",
@@ -180,8 +186,14 @@ func TestCmdListenerGenerate_InputToOptions(t *testing.T) {
 			expectedOutput:         "json",
 		},
 		{
-			name:                   "test2",
-			flags:                  common.CommandListenerGenerateFlags{"", "", "secret", "tcp", "yaml"},
+			name: "test2",
+			flags: common.CommandListenerGenerateFlags{
+				RoutingKey:     "",
+				Host:           "",
+				TlsCredentials: "secret",
+				ListenerType:   "tcp",
+				Output:         "yaml",
+			},
 			expectedTlsCredentials: "secret",
 			expectedHost:           "test2",
 			expectedRoutingKey:     "test2",
diff --git a/internal/cmd/skupper/listener/nonkube/listener_create_test.go b/internal/cmd/skupper/listener/nonkube/listener_create_test.go
index b4eed6bf..4b48d964 100644
--- a/internal/cmd/skupper/listener/nonkube/listener_create_test.go
+++ b/internal/cmd/skupper/listener/nonkube/listener_create_test.go
@@ -148,8 +148,15 @@ func TestNonKubeCmdListenerCreate_InputToOptions(t *testing.T) {
 
 	testTable := []test{
 		{
-			name:                   "test1",
-			flags:                  common.CommandListenerCreateFlags{"backend", "", "secret", "tcp", 0, "none"},
+			name: "test1",
+			flags: common.CommandListenerCreateFlags{
+				RoutingKey:     "backend",
+				Host:           "",
+				TlsCredentials: "secret",
+				ListenerType:   "tcp",
+				Timeout:        0,
+				Wait:           "none",
+			},
 			expectedTlsCredentials: "secret",
 			expectedHost:           "0.0.0.0",
 			expectedRoutingKey:     "backend",
@@ -157,9 +164,16 @@ func TestNonKubeCmdListenerCreate_InputToOptions(t *testing.T) {
 			expectedNamespace:      "default",
 		},
 		{
-			name:                   "test2",
-			namespace:              "test",
-			flags:                  common.CommandListenerCreateFlags{"backend", "1.2.3.4", "secret", "tcp", 0, "configured"},
+			name:      "test2",
+			namespace: "test",
+			flags: common.CommandListenerCreateFlags{
+				RoutingKey:     "backend",
+				Host:           "1.2.3.4",
+				TlsCredentials: "secret",
+				ListenerType:   "tcp",
+				Timeout:        0,
+				Wait:           "configured",
+			},
 			expectedTlsCredentials: "secret",
 			expectedHost:           "1.2.3.4",
 			expectedRoutingKey:     "backend",
@@ -167,9 +181,16 @@ func TestNonKubeCmdListenerCreate_InputToOptions(t *testing.T) {
 			expectedNamespace:      "test",
 		},
 		{
-			name:                   "test3",
-			namespace:              "default",
-			flags:                  common.CommandListenerCreateFlags{"", "", "secret", "tcp", 0, "ready"},
+			name:      "test3",
+			namespace: "default",
+			flags: common.CommandListenerCreateFlags{
+				RoutingKey:     "",
+				Host:           "",
+				TlsCredentials: "secret",
+				ListenerType:   "tcp",
+				Timeout:        0,
+				Wait:           "ready",
+			},
 			expectedTlsCredentials: "secret",
 			expectedHost:           "0.0.0.0",
 			expectedRoutingKey:     "my-listener",
diff --git a/internal/cmd/skupper/listener/nonkube/listener_generate_test.go b/internal/cmd/skupper/listener/nonkube/listener_generate_test.go
index dd25a657..5b004098 100644
--- a/internal/cmd/skupper/listener/nonkube/listener_generate_test.go
+++ b/internal/cmd/skupper/listener/nonkube/listener_generate_test.go
@@ -157,8 +157,13 @@ func TestNonKubeCmdListenerGenerate_InputToOptions(t *testing.T) {
 
 	testTable := []test{
 		{
-			name:                   "test1",
-			flags:                  common.CommandListenerGenerateFlags{"backend", "", "secret", "tcp", "json"},
+			name: "test1",
+			flags: common.CommandListenerGenerateFlags{
+				RoutingKey:     "backend",
+				Host:           "",
+				TlsCredentials: "secret",
+				ListenerType:   "tcp",
+				Output:         "json"},
 			expectedTlsCredentials: "secret",
 			expectedHost:           "0.0.0.0",
 			expectedRoutingKey:     "backend",
@@ -167,9 +172,15 @@ func TestNonKubeCmdListenerGenerate_InputToOptions(t *testing.T) {
 			expectedNamespace:      "default",
 		},
 		{
-			name:                   "test2",
-			namespace:              "test",
-			flags:                  common.CommandListenerGenerateFlags{"backend", "1.2.3.4", "secret", "tcp", "json"},
+			name:      "test2",
+			namespace: "test",
+			flags: common.CommandListenerGenerateFlags{
+				RoutingKey:     "backend",
+				Host:           "1.2.3.4",
+				TlsCredentials: "secret",
+				ListenerType:   "tcp",
+				Output:         "json",
+			},
 			expectedTlsCredentials: "secret",
 			expectedHost:           "1.2.3.4",
 			expectedRoutingKey:     "backend",
@@ -178,9 +189,14 @@ func TestNonKubeCmdListenerGenerate_InputToOptions(t *testing.T) {
 			expectedNamespace:      "test",
 		},
 		{
-			name:                   "test3",
-			namespace:              "default",
-			flags:                  common.CommandListenerGenerateFlags{"", "", "secret", "tcp", "yaml"},
+			name:      "test3",
+			namespace: "default",
+			flags: common.CommandListenerGenerateFlags{
+				RoutingKey:     "",
+				Host:           "",
+				TlsCredentials: "secret",
+				ListenerType:   "tcp",
+				Output:         "yaml"},
 			expectedTlsCredentials: "secret",
 			expectedHost:           "0.0.0.0",
 			expectedRoutingKey:     "my-listener",
diff --git a/internal/cmd/skupper/token/kube/token_redeem_test.go b/internal/cmd/skupper/token/kube/token_redeem_test.go
index ae8ee34d..d243daa8 100644
--- a/internal/cmd/skupper/token/kube/token_redeem_test.go
+++ b/internal/cmd/skupper/token/kube/token_redeem_test.go
@@ -449,11 +449,11 @@ func newCmdCreateAccessTokenFile(fileName string) error {
 	s := json.NewYAMLSerializer(json.DefaultMetaFactory, scheme.Scheme, scheme.Scheme)
 	out, err := os.Create(fileName)
 	if err != nil {
-		return fmt.Errorf("Could not write to file " + fileName + ": " + err.Error())
+		return fmt.Errorf("could not write to file %s:%s ", fileName, err.Error())
 	}
 	err = s.Encode(&resource, out)
 	if err != nil {
-		return fmt.Errorf("Could not write out generated token: " + err.Error())
+		return fmt.Errorf("could not write out generated token: %s", err.Error())
 	}
 
 	return nil
diff --git a/internal/nonkube/common/fs_config_renderer_test.go b/internal/nonkube/common/fs_config_renderer_test.go
index d47ebfaf..23514af5 100644
--- a/internal/nonkube/common/fs_config_renderer_test.go
+++ b/internal/nonkube/common/fs_config_renderer_test.go
@@ -34,7 +34,7 @@ func testFileSystemConfigurationRendererRender(t *testing.T, addInputCertificate
 		//assert.Assert(t, err)
 	}()
 	if addInputCertificates {
-		t.Logf(customOutputPath)
+		t.Log(customOutputPath)
 		createInputCertificates(t, customOutputPath)
 	}
 	fsConfigRenderer := new(FileSystemConfigurationRenderer)
diff --git a/tests/README.md b/tests/README.md
index c91f1c32..623e1343 100644
--- a/tests/README.md
+++ b/tests/README.md
@@ -24,6 +24,7 @@ tests/
 │    ├── hello-world/          # Basic Skupper functionality test
 │    ├── attached-connector/   # Network performance test with attached connectors
 │    ├── redis/                # Redis test
+│    ├── ha/                   # High availability test using Skupper
 └── README.md                  # This file
 ```
 
@@ -33,9 +34,10 @@ The `e2e` directory contains tests that validate Skupper functionality across di
 
 ### Available E2E Tests
 
-- **[hello-world](e2e/hello-world/)**: A simple test to verify basic Skupper functionality by deploying frontend and backend components across Skupper sites.
-- **[attached-connector](e2e/attached-connector/)**: A test to validate Skupper connectivity using attached connectors, including network performance testing with iperf3.
-- **[redis](e2e/redis/)**: A test to validate Skupper functionality with Redis, including data persistence and replication.
+- **[hello-world](e2e/scenarios/hello-world/)**: A simple test to verify basic Skupper functionality by deploying frontend and backend components across Skupper sites.
+- **[attached-connector](e2e/scenarios/attached-connector/)**: A test to validate Skupper connectivity using attached connectors, including network performance testing with iperf3.
+- **[redis](e2e/scenarios/redis/)**: A test to validate Skupper functionality with Redis, including data persistence and replication.
+- **[ha](e2e/scenarios/ha/)**: A test to validate Skupper's high availability mode.
 
 ## Adding a new E2E test to be run on CI
 
@@ -81,6 +83,26 @@ The Makefile will automatically:
 
 ### 2. Run Test
 
+**Prerequisites**
+
+* A cluster with Skupper installed cluster-wide
+* Your kubeconfig is set to that cluster
+
+If you are testing changes to Skupper code, create the images you want to test by running the following command in the project root folder:
+
+```
+# Under the project root folder
+make podman-build
+```
+
+**Tip:** To quickly create a [kind](https://kind.sigs.k8s.io/) cluster:
+
+```bash
+KUBECONFIG=~/.kube/config ../scripts/kind-dev-cluster -r --metallb -i podman
+```
+
+**Note:** The scripts create and delete namespaces.
+
 ```bash
 # Run a specific test
 make test TEST="hello-world"
@@ -167,6 +189,24 @@ e2e/scenarios/your-test/
 └── test.yml              # Main test playbook
 ```
 
+## Creating a New E2E Test
+
+To create a new end-to-end (E2E) test, it is recommended to use the `hello-world` test as a base. The `hello-world` test is the simplest and most basic test, making it an ideal starting point for new tests. Follow these steps:
+
+1. **Copy the Base Test**: Duplicate the `hello-world` directory located in `e2e/scenarios/`.
+
+   ```bash
+   cp -r tests/e2e/scenarios/hello-world tests/e2e/scenarios/your-new-test
+   ```
+
+2. **Modify the Test Name**: Rename the copied directory and update any references to `hello-world` within the test files to reflect the new test name.
+
+3. **Customize the Test**: Adjust the test logic, configuration, and any other necessary components to fit the new test scenario.
+
+4. **Document the Test**: Ensure that the new test directory includes a `README.md` file with instructions on how to run the test.
+
+By following these steps, you can efficiently create a new E2E test that integrates seamlessly with the existing test framework.
+
 ## License
 
 This project is licensed under the Apache License, Version 2.0. See the [LICENSE](../LICENSE) file for more details.