Repository: FoundationDB/fdb-kubernetes-operator
Pull request #2247: Ensure that pods are replaced if we change the Pod IP family
Author: login: johscheuer
Labels: name: bug
description: Something isn't working
Created: 2025-03-11T15:33:15Z | Merged: 2025-03-18T18:40:47Z

# Description

Fixes: https://github.com/FoundationDB/fdb-kubernetes-operator/issues/2243

## Type of change

- Bug fix (non-breaking change which fixes an issue)

## Discussion

With the previous behaviour it could happen that the coordinators went unreachable because they were listening on the wrong IP family and then the operator would get stuck. This change prevents it by spinning up a new set of pods.

## Testing

Added a new e2e test to convert from `nil` family to `v4`. I run some tests in a dual-stack cluster.

## Documentation

-

## Follow-up

https://github.com/FoundationDB/fdb-kubernetes-operator/issues/834 the idea here would be to listen on both addresses for some time.



Diff:
diff --git a/api/v1beta2/foundationdb_labels.go b/api/v1beta2/foundationdb_labels.go
index 9ed61e649..47ec16577 100644
--- a/api/v1beta2/foundationdb_labels.go
+++ b/api/v1beta2/foundationdb_labels.go
@@ -40,6 +40,9 @@ const (
 	// gets its public IP from.
 	PublicIPSourceAnnotation = "foundationdb.org/public-ip-source"
 
+	// IPFamilyAnnotation is an annotation key that specifies the IP family of the pod.
+	IPFamilyAnnotation = "foundationdb.org/ip-family"
+
 	// PublicIPAnnotation is an annotation key that specifies the current public
 	// IP for a pod.
 	PublicIPAnnotation = "foundationdb.org/public-ip"
diff --git a/api/v1beta2/foundationdbcluster_types.go b/api/v1beta2/foundationdbcluster_types.go
index 08f9904ff..2ec530fca 100644
--- a/api/v1beta2/foundationdbcluster_types.go
+++ b/api/v1beta2/foundationdbcluster_types.go
@@ -2104,8 +2104,7 @@ type RoutingConfig struct {
 
 	// PodIPFamily tells the pod which family of IP addresses to use.
 	// You can use 4 to represent IPv4, and 6 to represent IPv6.
-	// This feature is only supported in FDB 7.0 or later, and requires
-	// dual-stack support in your Kubernetes environment.
+	// This feature requires dual-stack support in your Kubernetes environment.
 	PodIPFamily *int `json:"podIPFamily,omitempty"`
 
 	// UseDNSInClusterFile determines whether to use DNS names rather than IP
@@ -2907,6 +2906,13 @@ func (cluster *FoundationDBCluster) Validate() error {
 		}
 	}
 
+	if cluster.Spec.Routing.PodIPFamily != nil {
+		ipFamily := *cluster.Spec.Routing.PodIPFamily
+		if ipFamily != 4 && ipFamily != 6 {
+			validations = append(validations, fmt.Sprintf("Pod IP Family %d is not valid, only 4 or 6 are allowed.", ipFamily))
+		}
+	}
+
 	if len(validations) == 0 {
 		return nil
 	}
@@ -3052,9 +3058,16 @@ func (cluster *FoundationDBCluster) GetProcessGroupID(processClass ProcessClass,
 	return fmt.Sprintf("%s-%s-%d", cluster.Name, processClass.GetProcessClassForPodName(), idNum), processGroupID
 }
 
+var (
+	// PodIPFamilyIPv4 represents the desired IP family as IPv4.
+	PodIPFamilyIPv4 = 4
+	// PodIPFamilyIPv6 represents the desired IP family as IPv6.
+	PodIPFamilyIPv6 = 6
+)
+
 // IsPodIPFamily6 determines whether the podIPFamily setting in cluster is set to use the IPv6 family.
 func (cluster *FoundationDBCluster) IsPodIPFamily6() bool {
-	return pointer.IntDeref(cluster.Spec.Routing.PodIPFamily, 4) == 6
+	return pointer.IntDeref(cluster.Spec.Routing.PodIPFamily, PodIPFamilyIPv4) == PodIPFamilyIPv6
 }
 
 // ProcessSharesDC returns true if the process's locality matches the cluster's Datacenter.
diff --git a/docs/cluster_spec.md b/docs/cluster_spec.md
index addbec14e..47de8fd72 100644
--- a/docs/cluster_spec.md
+++ b/docs/cluster_spec.md
@@ -468,7 +468,7 @@ RoutingConfig allows configuring routing to our pods, and services that sit in f
 | ----- | ----------- | ------ | -------- |
 | headlessService | Headless determines whether we want to run a headless service for the cluster. | *bool | false |
 | publicIPSource | PublicIPSource specifies what source a process should use to get its public IPs.  This supports the values `pod` and `service`. | *[PublicIPSource](#publicipsource) | false |
-| podIPFamily | PodIPFamily tells the pod which family of IP addresses to use. You can use 4 to represent IPv4, and 6 to represent IPv6. This feature is only supported in FDB 7.0 or later, and requires dual-stack support in your Kubernetes environment. | *int | false |
+| podIPFamily | PodIPFamily tells the pod which family of IP addresses to use. You can use 4 to represent IPv4, and 6 to represent IPv6. This feature requires dual-stack support in your Kubernetes environment. | *int | false |
 | useDNSInClusterFile | UseDNSInClusterFile determines whether to use DNS names rather than IP addresses to identify coordinators in the cluster file. This requires FoundationDB 7.0+. | *bool | false |
 | defineDNSLocalityFields | DefineDNSLocalityFields determines whether to define pod DNS names on pod specs and provide them in the locality arguments to fdbserver.  This is ignored if UseDNSInCluster is true. | *bool | false |
 | dnsDomain | DNSDomain defines the cluster domain used in a DNS name generated for a service. The default is `cluster.local`. | *string | false |
diff --git a/e2e/test_operator/operator_test.go b/e2e/test_operator/operator_test.go
index 7ec3fbc32..aed8e2877 100644
--- a/e2e/test_operator/operator_test.go
+++ b/e2e/test_operator/operator_test.go
@@ -36,6 +36,7 @@ import (
 	"fmt"
 	"log"
 	"math"
+	"strconv"
 	"strings"
 	"time"
 
@@ -2737,4 +2738,44 @@ var _ = Describe("Operator", Label("e2e", "pr"), func() {
 			})
 		})
 	})
+
+	When("the pod IP family is changed", func() {
+		var testStartTime time.Time
+		// TODO (johscheuer): Make the IP family configurable in the future.
+		var podIPFamily = fdbv1beta2.PodIPFamilyIPv4
+
+		BeforeEach(func() {
+			testStartTime = time.Now()
+			spec := fdbCluster.GetCluster().Spec.DeepCopy()
+			spec.Routing.PodIPFamily = pointer.Int(podIPFamily)
+			fdbCluster.UpdateClusterSpecWithSpec(spec)
+			Expect(fdbCluster.WaitForReconciliation()).To(Succeed())
+		})
+
+		AfterEach(func() {
+			spec := fdbCluster.GetCluster().Spec.DeepCopy()
+			spec.Routing.PodIPFamily = nil
+			fdbCluster.UpdateClusterSpecWithSpec(spec)
+			Expect(fdbCluster.WaitForReconciliation(fixtures.SoftReconcileOption(true))).To(Succeed())
+		})
+
+		It("should replace all pods and configure them properly", func() {
+			pods := fdbCluster.GetPods()
+			podIPFamilyString := strconv.Itoa(podIPFamily)
+
+			for _, pod := range pods.Items {
+				if !pod.DeletionTimestamp.IsZero() {
+					continue
+				}
+
+				if pod.Status.Phase != corev1.PodRunning {
+					log.Println("ignoring pod:", pod.Name, "with pod phase", pod.Status.Phase, "message:", pod.Status.Message)
+					continue
+				}
+
+				Expect(pod.CreationTimestamp.After(testStartTime)).To(BeTrue())
+				Expect(pod.Annotations).To(HaveKeyWithValue(fdbv1beta2.IPFamilyAnnotation, podIPFamilyString))
+			}
+		})
+	})
 })
diff --git a/internal/pod_helper.go b/internal/pod_helper.go
index 101f0282c..bdb69f751 100644
--- a/internal/pod_helper.go
+++ b/internal/pod_helper.go
@@ -25,72 +25,57 @@ import (
 	"encoding/hex"
 	"encoding/json"
 	"fmt"
+	monitorapi "github.com/apple/foundationdb/fdbkubernetesmonitor/api"
 	"net"
 	"strconv"
 
-	"github.com/go-logr/logr"
-
-	"k8s.io/utils/pointer"
-
 	fdbv1beta2 "github.com/FoundationDB/fdb-kubernetes-operator/v2/api/v1beta2"
+	"github.com/go-logr/logr"
 	corev1 "k8s.io/api/core/v1"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"k8s.io/utils/pointer"
 	"sigs.k8s.io/controller-runtime/pkg/client"
 )
 
 // GetPublicIPsForPod returns the public IPs for a Pod
 func GetPublicIPsForPod(pod *corev1.Pod, log logr.Logger) []string {
-	var podIPFamily *int
-
 	if pod == nil {
 		return []string{}
 	}
 
-	for _, container := range pod.Spec.Containers {
-		if container.Name != fdbv1beta2.SidecarContainerName {
-			continue
-		}
-		for indexOfArgument, argument := range container.Args {
-			if argument == "--public-ip-family" && indexOfArgument < len(container.Args)-1 {
-				familyString := container.Args[indexOfArgument+1]
-				family, err := strconv.Atoi(familyString)
-				if err != nil {
-					log.Error(err, "Error parsing public IP family", "family", familyString)
-					return nil
-				}
-				podIPFamily = &family
-				break
-			}
-		}
+	podIPFamily, err := GetIPFamily(pod)
+	if err != nil {
+		log.Error(err, "Could not parse IP family")
+		return []string{pod.Status.PodIP}
 	}
 
-	if podIPFamily != nil {
-		podIPs := pod.Status.PodIPs
-		matchingIPs := make([]string, 0, len(podIPs))
+	if podIPFamily == nil {
+		return []string{pod.Status.PodIP}
+	}
 
-		for _, podIP := range podIPs {
-			ip := net.ParseIP(podIP.IP)
-			if ip == nil {
-				log.Error(nil, "Failed to parse IP from pod", "ip", podIP)
-				continue
-			}
-			matches := false
-			switch *podIPFamily {
-			case 4:
-				matches = ip.To4() != nil
-			case 6:
-				matches = ip.To4() == nil
-			default:
-				log.Error(nil, "Could not match IP address against IP family", "family", *podIPFamily)
-			}
-			if matches {
-				matchingIPs = append(matchingIPs, podIP.IP)
-			}
+	podIPs := pod.Status.PodIPs
+	matchingIPs := make([]string, 0, len(podIPs))
+	for _, podIP := range podIPs {
+		ip := net.ParseIP(podIP.IP)
+		if ip == nil {
+			log.Error(nil, "Failed to parse IP from pod", "ip", podIP)
+			continue
+		}
+		matches := false
+		switch *podIPFamily {
+		case fdbv1beta2.PodIPFamilyIPv4:
+			matches = ip.To4() != nil
+		case fdbv1beta2.PodIPFamilyIPv6:
+			matches = ip.To4() == nil
+		default:
+			log.Error(nil, "Could not match IP address against IP family", "family", *podIPFamily)
+		}
+		if matches {
+			matchingIPs = append(matchingIPs, podIP.IP)
 		}
-		return matchingIPs
 	}
 
-	return []string{pod.Status.PodIP}
+	return matchingIPs
 }
 
 // GetProcessGroupIDFromMeta fetches the process group ID from an object's metadata.
@@ -237,6 +222,82 @@ func GetPublicIPSource(pod *corev1.Pod) (fdbv1beta2.PublicIPSource, error) {
 	return fdbv1beta2.PublicIPSource(source), nil
 }
 
+// getIPFamilyFromPod returns the IP family from the pod configuration.
+func getIPFamilyFromPod(pod *corev1.Pod) (*int, error) {
+	if GetImageType(pod) == fdbv1beta2.ImageTypeUnified {
+		currentData, present := pod.Annotations[monitorapi.CurrentConfigurationAnnotation]
+		if !present {
+			return nil, fmt.Errorf("could not read current launcher configuration")
+		}
+
+		currentConfiguration := monitorapi.ProcessConfiguration{}
+		err := json.Unmarshal([]byte(currentData), &currentConfiguration)
+		if err != nil {
+			return nil, fmt.Errorf("could not parse current process configuration: %w", err)
+		}
+
+		for _, argument := range currentConfiguration.Arguments {
+			if argument.ArgumentType != monitorapi.IPListArgumentType {
+				continue
+			}
+
+			return validateIPFamily(argument.IPFamily)
+		}
+
+		// No IP List setting is defined.
+		return nil, nil
+	}
+
+	for _, container := range pod.Spec.Containers {
+		if container.Name != fdbv1beta2.SidecarContainerName {
+			continue
+		}
+
+		for indexOfArgument, argument := range container.Args {
+			if argument == "--public-ip-family" && indexOfArgument < len(container.Args)-1 {
+				return parseIPFamily(container.Args[indexOfArgument+1])
+			}
+		}
+	}
+
+	return nil, nil
+}
+
+// validateIPFamily will validate that the IP family is valid and return a pointer.
+func validateIPFamily(ipFamily int) (*int, error) {
+	if ipFamily != fdbv1beta2.PodIPFamilyIPv4 && ipFamily != fdbv1beta2.PodIPFamilyIPv6 {
+		return nil, fmt.Errorf("unsupported IP family %d", ipFamily)
+	}
+
+	return pointer.Int(ipFamily), nil
+}
+
+// parseIPFamily will convert a string into the IP family (int pointer) and validate that the value is a valid IP family.
+func parseIPFamily(ipFamilyValue string) (*int, error) {
+	ipFamily, err := strconv.Atoi(ipFamilyValue)
+	if err != nil {
+		return nil, err
+	}
+
+	return validateIPFamily(ipFamily)
+}
+
+// GetIPFamily determines the IP family based on the annotation. If the annotation is not present the method will try to
+// get the ip family based on the pod spec.
+func GetIPFamily(pod *corev1.Pod) (*int, error) {
+	if pod == nil {
+		return nil, fmt.Errorf("failed to fetch IP family from nil Pod")
+	}
+
+	ipFamilyValue, ok := pod.ObjectMeta.Annotations[fdbv1beta2.IPFamilyAnnotation]
+	if ipFamilyValue != "" && ok {
+		return parseIPFamily(ipFamilyValue)
+	}
+
+	// If the annotation is missing, try to get the pod IP family from the pod spec.
+	return getIPFamilyFromPod(pod)
+}
+
 // PodHasSidecarTLS determines whether a pod currently has TLS enabled for the sidecar process.
 // This method should only be used for split images.
 func PodHasSidecarTLS(pod *corev1.Pod) bool {
diff --git a/internal/pod_helper_test.go b/internal/pod_helper_test.go
new file mode 100644
index 000000000..9c8cc1486
--- /dev/null
+++ b/internal/pod_helper_test.go
@@ -0,0 +1,194 @@
+/*
+ * pod_helper_test.go
+ *
+ * This source file is part of the FoundationDB open source project
+ *
+ * Copyright 2018-2025 Apple Inc. and the FoundationDB project authors
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package internal
+
+import (
+	"errors"
+
+	fdbv1beta2 "github.com/FoundationDB/fdb-kubernetes-operator/v2/api/v1beta2"
+	monitorapi "github.com/apple/foundationdb/fdbkubernetesmonitor/api"
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+	corev1 "k8s.io/api/core/v1"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"k8s.io/utils/pointer"
+)
+
+var _ = Describe("pod_helper", func() {
+	DescribeTable("when getting the IP family from a pod", func(pod *corev1.Pod, expected *int, expectedErr error) {
+		result, err := GetIPFamily(pod)
+		if expectedErr != nil {
+			Expect(err).To(MatchError(expectedErr))
+		} else {
+			Expect(err).To(Succeed())
+		}
+		Expect(result).To(Equal(expected))
+	},
+		Entry("empty pod",
+			nil,
+			nil,
+			errors.New("failed to fetch IP family from nil Pod"),
+		),
+		Entry("pod has IP family annotation for IPv4",
+			&corev1.Pod{
+				ObjectMeta: metav1.ObjectMeta{
+					Annotations: map[string]string{
+						fdbv1beta2.IPFamilyAnnotation: "4",
+					},
+				},
+			},
+			pointer.Int(fdbv1beta2.PodIPFamilyIPv4),
+			nil,
+		),
+		Entry("pod has IP family annotation for IPv6",
+			&corev1.Pod{
+				ObjectMeta: metav1.ObjectMeta{
+					Annotations: map[string]string{
+						fdbv1beta2.IPFamilyAnnotation: "6",
+					},
+				},
+			},
+			pointer.Int(fdbv1beta2.PodIPFamilyIPv6),
+			nil,
+		),
+		Entry("pod has IP family annotation with invalid value",
+			&corev1.Pod{
+				ObjectMeta: metav1.ObjectMeta{
+					Annotations: map[string]string{
+						fdbv1beta2.IPFamilyAnnotation: "1337",
+					},
+				},
+			},
+			nil,
+			errors.New("unsupported IP family 1337"),
+		),
+		Entry("pod has no IP family annotation and nothing specified in the pod spec",
+			&corev1.Pod{
+				ObjectMeta: metav1.ObjectMeta{},
+			},
+			nil,
+			nil,
+		),
+		Entry("pod has no IP family annotation and uses IP family IPv4 with split image",
+			&corev1.Pod{
+				ObjectMeta: metav1.ObjectMeta{
+					Annotations: map[string]string{
+						fdbv1beta2.ImageTypeAnnotation: string(fdbv1beta2.ImageTypeSplit),
+					},
+				},
+				Spec: corev1.PodSpec{
+					Containers: []corev1.Container{
+						{
+							Name: fdbv1beta2.SidecarContainerName,
+							Args: []string{
+								"--public-ip-family",
+								"4",
+							},
+						},
+					},
+				},
+			},
+			pointer.Int(fdbv1beta2.PodIPFamilyIPv4),
+			nil,
+		),
+		Entry("pod has no IP family annotation and uses IP family IPv6 with split image",
+			&corev1.Pod{
+				ObjectMeta: metav1.ObjectMeta{
+					Annotations: map[string]string{
+						fdbv1beta2.ImageTypeAnnotation: string(fdbv1beta2.ImageTypeSplit),
+					},
+				},
+				Spec: corev1.PodSpec{
+					Containers: []corev1.Container{
+						{
+							Name: fdbv1beta2.SidecarContainerName,
+							Args: []string{
+								"--public-ip-family",
+								"6",
+							},
+						},
+					},
+				},
+			},
+			pointer.Int(fdbv1beta2.PodIPFamilyIPv6),
+			nil,
+		),
+		Entry("pod has no IP family annotation and uses unsupported IP family with split image",
+			&corev1.Pod{
+				ObjectMeta: metav1.ObjectMeta{
+					Annotations: map[string]string{
+						fdbv1beta2.ImageTypeAnnotation: string(fdbv1beta2.ImageTypeSplit),
+					},
+				},
+				Spec: corev1.PodSpec{
+					Containers: []corev1.Container{
+						{
+							Name: fdbv1beta2.SidecarContainerName,
+							Args: []string{
+								"--public-ip-family",
+								"1337",
+							},
+						},
+					},
+				},
+			},
+			nil,
+			errors.New("unsupported IP family 1337"),
+		),
+
+		Entry("pod has no IP family annotation and uses IP family IPv4 with unified image",
+			&corev1.Pod{
+				ObjectMeta: metav1.ObjectMeta{
+					Annotations: map[string]string{
+						fdbv1beta2.ImageTypeAnnotation:            string(fdbv1beta2.ImageTypeUnified),
+						monitorapi.CurrentConfigurationAnnotation: "{\"arguments\": [{\"type\": \"IPList\", \"ipFamily\": 4}]}",
+					},
+				},
+			},
+			pointer.Int(fdbv1beta2.PodIPFamilyIPv4),
+			nil,
+		),
+		Entry("pod has no IP family annotation and uses IP family IPv6 with unified image",
+			&corev1.Pod{
+				ObjectMeta: metav1.ObjectMeta{
+					Annotations: map[string]string{
+						fdbv1beta2.ImageTypeAnnotation:            string(fdbv1beta2.ImageTypeUnified),
+						monitorapi.CurrentConfigurationAnnotation: "{\"arguments\": [{\"type\": \"IPList\", \"ipFamily\": 6}]}",
+					},
+				},
+			},
+			pointer.Int(fdbv1beta2.PodIPFamilyIPv6),
+			nil,
+		),
+		Entry("pod has no IP family annotation and uses unsupported IP family with unified image",
+			&corev1.Pod{
+				ObjectMeta: metav1.ObjectMeta{
+					Annotations: map[string]string{
+						fdbv1beta2.ImageTypeAnnotation:            string(fdbv1beta2.ImageTypeUnified),
+						monitorapi.CurrentConfigurationAnnotation: "{\"arguments\": [{\"type\": \"IPList\", \"ipFamily\": 1337}]}",
+					},
+				},
+			},
+			nil,
+			errors.New("unsupported IP family 1337"),
+		),
+	)
+})
diff --git a/internal/pod_models.go b/internal/pod_models.go
index 3edfe4fec..553ab93c0 100644
--- a/internal/pod_models.go
+++ b/internal/pod_models.go
@@ -1039,6 +1039,9 @@ func GetPodMetadata(cluster *fdbv1beta2.FoundationDBCluster, processClass fdbv1b
 	metadata.Annotations[fdbv1beta2.LastSpecKey] = specHash
 	metadata.Annotations[fdbv1beta2.PublicIPSourceAnnotation] = string(cluster.GetPublicIPSource())
 	metadata.Annotations[fdbv1beta2.ImageTypeAnnotation] = string(cluster.DesiredImageType())
+	if cluster.Spec.Routing.PodIPFamily != nil {
+		metadata.Annotations[fdbv1beta2.IPFamilyAnnotation] = strconv.Itoa(*cluster.Spec.Routing.PodIPFamily)
+	}
 
 	return metadata
 }
diff --git a/internal/replacements/replacements.go b/internal/replacements/replacements.go
index 9884583b0..17558d17d 100644
--- a/internal/replacements/replacements.go
+++ b/internal/replacements/replacements.go
@@ -234,6 +234,21 @@ func processGroupNeedsRemovalForPod(cluster *fdbv1beta2.FoundationDBCluster, pod
 		return true, nil
 	}
 
+	podIPFamily, err := internal.GetIPFamily(pod)
+	if err != nil {
+		return false, err
+	}
+
+	if !equality.Semantic.DeepEqual(cluster.Spec.Routing.PodIPFamily, podIPFamily) {
+		logger.Info("Replace process group",
+			"reason", "pod IP family has changed",
+			"currentPodIPFamily", podIPFamily,
+			"desiredPodIPFamily", cluster.Spec.Routing.PodIPFamily,
+		)
+
+		return true, nil
+	}
+
 	if cluster.NeedsReplacement(processGroup) {
 		jsonSpec, err := json.Marshal(spec)
 		if err != nil {
diff --git a/internal/replacements/replacements_test.go b/internal/replacements/replacements_test.go
index b73586351..b8eb4a921 100644
--- a/internal/replacements/replacements_test.go
+++ b/internal/replacements/replacements_test.go
@@ -3,7 +3,7 @@
  *
  * This source file is part of the FoundationDB open source project
  *
- * Copyright 2021-2024 Apple Inc. and the FoundationDB project authors
+ * Copyright 2021-2025 Apple Inc. and the FoundationDB project authors
  *
  * Licensed under the Apache License, Version 2.0 (the "License");
  * you may not use this file except in compliance with the License.
@@ -22,25 +22,23 @@ package replacements
 
 import (
 	"context"
+	"encoding/json"
 	"fmt"
 
-	"github.com/FoundationDB/fdb-kubernetes-operator/v2/pkg/podmanager"
-	ctrlClient "sigs.k8s.io/controller-runtime/pkg/client"
-
-	logf "sigs.k8s.io/controller-runtime/pkg/log"
-	"sigs.k8s.io/controller-runtime/pkg/log/zap"
-
-	"k8s.io/utils/pointer"
-
+	fdbv1beta2 "github.com/FoundationDB/fdb-kubernetes-operator/v2/api/v1beta2"
 	"github.com/FoundationDB/fdb-kubernetes-operator/v2/internal"
+	"github.com/FoundationDB/fdb-kubernetes-operator/v2/pkg/podmanager"
+	monitorapi "github.com/apple/foundationdb/fdbkubernetesmonitor/api"
 	"github.com/go-logr/logr"
-
-	fdbv1beta2 "github.com/FoundationDB/fdb-kubernetes-operator/v2/api/v1beta2"
 	. "github.com/onsi/ginkgo/v2"
 	. "github.com/onsi/gomega"
 	corev1 "k8s.io/api/core/v1"
 	"k8s.io/apimachinery/pkg/api/resource"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"k8s.io/utils/pointer"
+	ctrlClient "sigs.k8s.io/controller-runtime/pkg/client"
+	logf "sigs.k8s.io/controller-runtime/pkg/log"
+	"sigs.k8s.io/controller-runtime/pkg/log/zap"
 )
 
 var _ = Describe("replace_misconfigured_pods", func() {
@@ -533,6 +531,309 @@ var _ = Describe("replace_misconfigured_pods", func() {
 					})
 				})
 			})
+
+			When("the pod IP family is checked", func() {
+				When("the pod IP family matches", func() {
+					When("the pod ip family is nil", func() {
+						It("should *not* need a removal", func() {
+							Expect(needsRemoval).To(BeFalse())
+							Expect(err).NotTo(HaveOccurred())
+						})
+					})
+
+					When("the pod ip family is v4", func() {
+						BeforeEach(func() {
+							cluster.Spec.Routing.PodIPFamily = pointer.Int(fdbv1beta2.PodIPFamilyIPv4)
+						})
+
+						When("the image type is split", func() {
+							BeforeEach(func() {
+								for idx, container := range pod.Spec.Containers {
+									if container.Name != fdbv1beta2.SidecarContainerName {
+										continue
+									}
+
+									pod.Spec.Containers[idx].Args = append(pod.Spec.Containers[idx].Args, "--public-ip-family", "4")
+								}
+							})
+
+							It("should *not* need a removal", func() {
+								Expect(needsRemoval).To(BeFalse())
+								Expect(err).NotTo(HaveOccurred())
+							})
+						})
+
+						When("the image type is unified", func() {
+							BeforeEach(func() {
+								unified := fdbv1beta2.ImageTypeUnified
+								cluster.Spec.ImageType = &unified
+								pod.Annotations[fdbv1beta2.ImageTypeAnnotation] = string(fdbv1beta2.ImageTypeUnified)
+
+								currentConfiguration := monitorapi.ProcessConfiguration{
+									Arguments: []monitorapi.Argument{
+										{
+											ArgumentType: monitorapi.IPListArgumentType,
+											IPFamily:     4,
+										},
+									},
+								}
+
+								currentConfig, err := json.Marshal(currentConfiguration)
+								Expect(err).NotTo(HaveOccurred())
+
+								pod.Annotations[monitorapi.CurrentConfigurationAnnotation] = string(currentConfig)
+							})
+
+							It("should *not* need a removal", func() {
+								Expect(needsRemoval).To(BeFalse())
+								Expect(err).NotTo(HaveOccurred())
+							})
+						})
+					})
+
+					When("the pod ip family is v6", func() {
+						BeforeEach(func() {
+							cluster.Spec.Routing.PodIPFamily = pointer.Int(fdbv1beta2.PodIPFamilyIPv6)
+						})
+
+						When("the image type is split", func() {
+							BeforeEach(func() {
+								for idx, container := range pod.Spec.Containers {
+									if container.Name != fdbv1beta2.SidecarContainerName {
+										continue
+									}
+
+									pod.Spec.Containers[idx].Args = append(pod.Spec.Containers[idx].Args, "--public-ip-family", "6")
+								}
+							})
+
+							It("should *not* need a removal", func() {
+								Expect(needsRemoval).To(BeFalse())
+								Expect(err).NotTo(HaveOccurred())
+							})
+						})
+
+						When("the image type is unified", func() {
+							BeforeEach(func() {
+								unified := fdbv1beta2.ImageTypeUnified
+								cluster.Spec.ImageType = &unified
+								pod.Annotations[fdbv1beta2.ImageTypeAnnotation] = string(fdbv1beta2.ImageTypeUnified)
+
+								currentConfiguration := monitorapi.ProcessConfiguration{
+									Arguments: []monitorapi.Argument{
+										{
+											ArgumentType: monitorapi.IPListArgumentType,
+											IPFamily:     6,
+										},
+									},
+								}
+
+								currentConfig, err := json.Marshal(currentConfiguration)
+								Expect(err).NotTo(HaveOccurred())
+
+								pod.Annotations[monitorapi.CurrentConfigurationAnnotation] = string(currentConfig)
+							})
+
+							It("should *not* need a removal", func() {
+								Expect(needsRemoval).To(BeFalse())
+								Expect(err).NotTo(HaveOccurred())
+							})
+						})
+					})
+				})
+
+				When("the pod IP family doesn't match", func() {
+					When("the pod ip family at the pod is nil", func() {
+						When("the pod ip family at cluster level is v4", func() {
+							BeforeEach(func() {
+								cluster.Spec.Routing.PodIPFamily = pointer.Int(fdbv1beta2.PodIPFamilyIPv4)
+							})
+
+							It("should need a removal", func() {
+								Expect(needsRemoval).To(BeTrue())
+								Expect(err).NotTo(HaveOccurred())
+							})
+						})
+
+						When("the pod ip family at cluster level is v6", func() {
+							BeforeEach(func() {
+								cluster.Spec.Routing.PodIPFamily = pointer.Int(fdbv1beta2.PodIPFamilyIPv4)
+							})
+
+							It("should need a removal", func() {
+								Expect(needsRemoval).To(BeTrue())
+								Expect(err).NotTo(HaveOccurred())
+							})
+						})
+					})
+
+					When("the pod ip family at the pod is v4", func() {
+						When("the image type is split", func() {
+							BeforeEach(func() {
+								for idx, container := range pod.Spec.Containers {
+									if container.Name != fdbv1beta2.SidecarContainerName {
+										continue
+									}
+
+									pod.Spec.Containers[idx].Args = append(pod.Spec.Containers[idx].Args, "--public-ip-family", "4")
+								}
+
+								pod.ObjectMeta.Annotations[fdbv1beta2.LastSpecKey] = ""
+							})
+
+							When("the pod ip family at cluster level is nil", func() {
+								BeforeEach(func() {
+									cluster.Spec.Routing.PodIPFamily = nil
+								})
+
+								It("should need a removal", func() {
+									Expect(needsRemoval).To(BeTrue())
+									Expect(err).NotTo(HaveOccurred())
+								})
+							})
+
+							When("the pod ip family at cluster level is v6", func() {
+								BeforeEach(func() {
+									cluster.Spec.Routing.PodIPFamily = pointer.Int(fdbv1beta2.PodIPFamilyIPv6)
+								})
+
+								It("should need a removal", func() {
+									Expect(needsRemoval).To(BeTrue())
+									Expect(err).NotTo(HaveOccurred())
+								})
+							})
+						})
+
+						When("the image type is unified", func() {
+							BeforeEach(func() {
+								unified := fdbv1beta2.ImageTypeUnified
+								cluster.Spec.ImageType = &unified
+								pod.Annotations[fdbv1beta2.ImageTypeAnnotation] = string(fdbv1beta2.ImageTypeUnified)
+
+								currentConfiguration := monitorapi.ProcessConfiguration{
+									Arguments: []monitorapi.Argument{
+										{
+											ArgumentType: monitorapi.IPListArgumentType,
+											IPFamily:     4,
+										},
+									},
+								}
+
+								currentConfig, err := json.Marshal(currentConfiguration)
+								Expect(err).NotTo(HaveOccurred())
+
+								pod.Annotations[monitorapi.CurrentConfigurationAnnotation] = string(currentConfig)
+								pod.ObjectMeta.Annotations[fdbv1beta2.LastSpecKey] = ""
+							})
+
+							When("the pod ip family at cluster level is nil", func() {
+								BeforeEach(func() {
+									cluster.Spec.Routing.PodIPFamily = nil
+								})
+
+								It("should need a removal", func() {
+									Expect(needsRemoval).To(BeTrue())
+									Expect(err).NotTo(HaveOccurred())
+								})
+							})
+
+							When("the pod ip family at cluster level is v6", func() {
+								BeforeEach(func() {
+									cluster.Spec.Routing.PodIPFamily = pointer.Int(fdbv1beta2.PodIPFamilyIPv6)
+								})
+
+								It("should need a removal", func() {
+									Expect(needsRemoval).To(BeTrue())
+									Expect(err).NotTo(HaveOccurred())
+								})
+							})
+						})
+					})
+
+					When("the pod ip family at the pod is v6", func() {
+						When("the image type is split", func() {
+							BeforeEach(func() {
+								for idx, container := range pod.Spec.Containers {
+									if container.Name != fdbv1beta2.SidecarContainerName {
+										continue
+									}
+
+									pod.Spec.Containers[idx].Args = append(pod.Spec.Containers[idx].Args, "--public-ip-family", "6")
+								}
+
+								pod.ObjectMeta.Annotations[fdbv1beta2.LastSpecKey] = ""
+							})
+
+							When("the pod ip family at cluster level is nil", func() {
+								BeforeEach(func() {
+									cluster.Spec.Routing.PodIPFamily = nil
+								})
+
+								It("should need a removal", func() {
+									Expect(needsRemoval).To(BeTrue())
+									Expect(err).NotTo(HaveOccurred())
+								})
+							})
+
+							When("the pod ip family at cluster level is v4", func() {
+								BeforeEach(func() {
+									cluster.Spec.Routing.PodIPFamily = pointer.Int(fdbv1beta2.PodIPFamilyIPv4)
+								})
+
+								It("should need a removal", func() {
+									Expect(needsRemoval).To(BeTrue())
+									Expect(err).NotTo(HaveOccurred())
+								})
+							})
+						})
+
+						When("the image type is unified", func() {
+							BeforeEach(func() {
+								unified := fdbv1beta2.ImageTypeUnified
+								cluster.Spec.ImageType = &unified
+								pod.Annotations[fdbv1beta2.ImageTypeAnnotation] = string(fdbv1beta2.ImageTypeUnified)
+
+								currentConfiguration := monitorapi.ProcessConfiguration{
+									Arguments: []monitorapi.Argument{
+										{
+											ArgumentType: monitorapi.IPListArgumentType,
+											IPFamily:     6,
+										},
+									},
+								}
+
+								currentConfig, err := json.Marshal(currentConfiguration)
+								Expect(err).NotTo(HaveOccurred())
+
+								pod.Annotations[monitorapi.CurrentConfigurationAnnotation] = string(currentConfig)
+								pod.ObjectMeta.Annotations[fdbv1beta2.LastSpecKey] = ""
+							})
+
+							When("the pod ip family at cluster level is nil", func() {
+								BeforeEach(func() {
+									cluster.Spec.Routing.PodIPFamily = nil
+								})
+
+								It("should need a removal", func() {
+									Expect(needsRemoval).To(BeTrue())
+									Expect(err).NotTo(HaveOccurred())
+								})
+							})
+
+							When("the pod ip family at cluster level is v4", func() {
+								BeforeEach(func() {
+									cluster.Spec.Routing.PodIPFamily = pointer.Int(fdbv1beta2.PodIPFamilyIPv4)
+								})
+
+								It("should need a removal", func() {
+									Expect(needsRemoval).To(BeTrue())
+									Expect(err).NotTo(HaveOccurred())
+								})
+							})
+						})
+					})
+				})
+			})
 		})
 
 		When("a log Pod is checked", func() {


Reviews:
user: login: brownleej
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-13T19:22:19Z
user: login: johscheuer
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-14T06:42:45Z
user: login: brownleej
body: 👍
state: APPROVED
author_association: MEMBER
submitted_at: 2025-03-18T16:27:44Z

Review comments:
diff_hunk: @@ -1039,6 +1039,9 @@ func GetPodMetadata(cluster *fdbv1beta2.FoundationDBCluster, processClass fdbv1b
 	metadata.Annotations[fdbv1beta2.LastSpecKey] = specHash
 	metadata.Annotations[fdbv1beta2.PublicIPSourceAnnotation] = string(cluster.GetPublicIPSource())
 	metadata.Annotations[fdbv1beta2.ImageTypeAnnotation] = string(cluster.DesiredImageType())
+	if cluster.Spec.Routing.PodIPFamily != nil {
+		metadata.Annotations[fdbv1beta2.IPFamilyAnnotation] = strconv.Itoa(*cluster.Spec.Routing.PodIPFamily)
path: internal/pod_models.go
user: login: brownleej
body: This looks like it will apply this annotation to all clusters with a PodIPFamily defined as soon as the operator is deployed. Do we expect this to be a fast and non-disruptive operation?
created_at: 2025-03-13T19:19:13Z
author_association: MEMBER
diff_hunk: @@ -237,6 +222,73 @@ func GetPublicIPSource(pod *corev1.Pod) (fdbv1beta2.PublicIPSource, error) {
 	return fdbv1beta2.PublicIPSource(source), nil
 }
 
+// GetIPFamilyFromPod returns the IP family from the pod configuration.
+func GetIPFamilyFromPod(pod *corev1.Pod) (*int, error) {
+	if GetImageType(pod) == fdbv1beta2.ImageTypeUnified {
+		currentData, present := pod.Annotations[monitorapi.CurrentConfigurationAnnotation]
+		if !present {
+			return nil, fmt.Errorf("could not read current launcher configruations")
+		}
+
+		currentConfiguration := monitorapi.ProcessConfiguration{}
+		err := json.Unmarshal([]byte(currentData), &currentConfiguration)
+		if err != nil {
+			return nil, fmt.Errorf("could not parse current process configuration: %w", err)
+		}
+
+		for _, argument := range currentConfiguration.Arguments {
+			if argument.ArgumentType != monitorapi.IPListArgumentType {
+				continue
+			}
+
+			return pointer.Int(argument.IPFamily), nil
+		}
+
+		// No IP List setting is defined.
+		return nil, nil
+	}
+
+	for _, container := range pod.Spec.Containers {
+		if container.Name != fdbv1beta2.SidecarContainerName {
+			continue
+		}
+
+		for indexOfArgument, argument := range container.Args {
+			if argument == "--public-ip-family" && indexOfArgument < len(container.Args)-1 {
+				familyString := container.Args[indexOfArgument+1]
+				family, err := strconv.Atoi(familyString)
+				if err == nil {
+					return &family, nil
+				}
+				break
+			}
+		}
+	}
+
+	return nil, nil
+}
+
+// GetIPFamily determines the IP family based on the annotation.
+// TODO (johscheuer): Make use of this method once we did the next release 2.2.0. This will ensure that the operator
+// can add the fdbv1beta2.IPFamilyAnnotation to all pods.
+//func GetIPFamily(pod *corev1.Pod) (*int, error) {
path: internal/pod_helper.go
user: login: brownleej
body: Is the intention that this function will be used in place of `GetIPFamilyFromPod` once we apply all the necessary annotations? What will happen if a cluster does not get re-reconciled between releases, and is left without the annotations for the IP family?
created_at: 2025-03-13T19:21:33Z
author_association: MEMBER
diff_hunk: @@ -1039,6 +1039,9 @@ func GetPodMetadata(cluster *fdbv1beta2.FoundationDBCluster, processClass fdbv1b
 	metadata.Annotations[fdbv1beta2.LastSpecKey] = specHash
 	metadata.Annotations[fdbv1beta2.PublicIPSourceAnnotation] = string(cluster.GetPublicIPSource())
 	metadata.Annotations[fdbv1beta2.ImageTypeAnnotation] = string(cluster.DesiredImageType())
+	if cluster.Spec.Routing.PodIPFamily != nil {
+		metadata.Annotations[fdbv1beta2.IPFamilyAnnotation] = strconv.Itoa(*cluster.Spec.Routing.PodIPFamily)
path: internal/pod_models.go
user: login: johscheuer
body: This operation will be non-disruptive and fast, as the operator will only update the pod metadata.
created_at: 2025-03-14T06:41:17Z
author_association: MEMBER
diff_hunk: @@ -237,6 +222,73 @@ func GetPublicIPSource(pod *corev1.Pod) (fdbv1beta2.PublicIPSource, error) {
 	return fdbv1beta2.PublicIPSource(source), nil
 }
 
+// GetIPFamilyFromPod returns the IP family from the pod configuration.
+func GetIPFamilyFromPod(pod *corev1.Pod) (*int, error) {
+	if GetImageType(pod) == fdbv1beta2.ImageTypeUnified {
+		currentData, present := pod.Annotations[monitorapi.CurrentConfigurationAnnotation]
+		if !present {
+			return nil, fmt.Errorf("could not read current launcher configruations")
+		}
+
+		currentConfiguration := monitorapi.ProcessConfiguration{}
+		err := json.Unmarshal([]byte(currentData), &currentConfiguration)
+		if err != nil {
+			return nil, fmt.Errorf("could not parse current process configuration: %w", err)
+		}
+
+		for _, argument := range currentConfiguration.Arguments {
+			if argument.ArgumentType != monitorapi.IPListArgumentType {
+				continue
+			}
+
+			return pointer.Int(argument.IPFamily), nil
+		}
+
+		// No IP List setting is defined.
+		return nil, nil
+	}
+
+	for _, container := range pod.Spec.Containers {
+		if container.Name != fdbv1beta2.SidecarContainerName {
+			continue
+		}
+
+		for indexOfArgument, argument := range container.Args {
+			if argument == "--public-ip-family" && indexOfArgument < len(container.Args)-1 {
+				familyString := container.Args[indexOfArgument+1]
+				family, err := strconv.Atoi(familyString)
+				if err == nil {
+					return &family, nil
+				}
+				break
+			}
+		}
+	}
+
+	return nil, nil
+}
+
+// GetIPFamily determines the IP family based on the annotation.
+// TODO (johscheuer): Make use of this method once we did the next release 2.2.0. This will ensure that the operator
+// can add the fdbv1beta2.IPFamilyAnnotation to all pods.
+//func GetIPFamily(pod *corev1.Pod) (*int, error) {
path: internal/pod_helper.go
user: login: johscheuer
body: That was the initial intention, I think a better approach is to use the value of the annotation if present and if not fall back to the logic from `GetIPFamilyFromPod`. I make the code adjustments later.
created_at: 2025-03-14T06:42:39Z
author_association: MEMBER

PR comments:
user: login: foundationdb-ci
created_at: 2025-03-11T18:43:55Z
body: 
### Result of fdb-kubernetes-operator-pr on Linux CentOS 7

* Commit ID: de04c61c738aae6d6554cd6b9c7770c198f01e18
* Duration 3:10:31
* Result: :x: **FAILED**
* Error: `Error while executing command: if $fail_test; then exit 1; fi. Reason: exit status 1`
* [Build Log](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator/fdb-kubernetes-operator-pr/716c9574-dfc8-4681-8b13-ac389c56c61f.gz) terminal output (available for 30 days)
* [Build Workspace](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator-pr/716c9574-dfc8-4681-8b13-ac389c56c61f/workspace.zip) zip file of the working directory (available for 30 days)

author_association: CONTRIBUTOR
user: login: johscheuer
created_at: 2025-03-12T10:31:19Z
body: > ### Result of fdb-kubernetes-operator-pr on Linux CentOS 7
> * Commit ID: [de04c61](https://github.com/FoundationDB/fdb-kubernetes-operator/commit/de04c61c738aae6d6554cd6b9c7770c198f01e18)
> * Duration 3:10:31
> * Result: ❌ **FAILED**
> * Error: `Error while executing command: if $fail_test; then exit 1; fi. Reason: exit status 1`
> * [Build Log](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator/fdb-kubernetes-operator-pr/716c9574-dfc8-4681-8b13-ac389c56c61f.gz) terminal output (available for 30 days)
> * [Build Workspace](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator-pr/716c9574-dfc8-4681-8b13-ac389c56c61f/workspace.zip) zip file of the working directory (available for 30 days)

The test failed because there was a pod with a `failed` status:

```text
operator-test-246my2ej-transaction-20242                    |0/2   |Failed  |-             |0        |7.1.57     |7.1.57        |               |ip-100-80-158-24.us-west-2.compute.internal  |11m
```
author_association: MEMBER
user: login: foundationdb-ci
created_at: 2025-03-12T13:48:01Z
body: 
### Result of fdb-kubernetes-operator-pr on Linux CentOS 7

* Commit ID: de04c61c738aae6d6554cd6b9c7770c198f01e18
* Duration 3:18:28
* Result: :x: **FAILED**
* Error: `Error while executing command: if $fail_test; then exit 1; fi. Reason: exit status 1`
* [Build Log](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator/fdb-kubernetes-operator-pr/73638e4e-4500-4971-a118-6c01d24e9f3b.gz) terminal output (available for 30 days)
* [Build Workspace](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator-pr/73638e4e-4500-4971-a118-6c01d24e9f3b/workspace.zip) zip file of the working directory (available for 30 days)

author_association: CONTRIBUTOR
user: login: foundationdb-ci
created_at: 2025-03-12T14:16:47Z
body: 
### Result of fdb-kubernetes-operator-pr on Linux CentOS 7

* Commit ID: abfd29c7bb08111ac3b18953b0cebb850e40bc4d
* Duration 3:37:29
* Result: :x: **FAILED**
* Error: `Error while executing command: if $fail_test; then exit 1; fi. Reason: exit status 1`
* [Build Log](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator/fdb-kubernetes-operator-pr/a012dd42-c045-4081-8344-56a7a9122197.gz) terminal output (available for 30 days)
* [Build Workspace](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator-pr/a012dd42-c045-4081-8344-56a7a9122197/workspace.zip) zip file of the working directory (available for 30 days)

author_association: CONTRIBUTOR
user: login: johscheuer
created_at: 2025-03-12T15:30:50Z
body: > ### Result of fdb-kubernetes-operator-pr on Linux CentOS 7
> * Commit ID: [abfd29c](https://github.com/FoundationDB/fdb-kubernetes-operator/commit/abfd29c7bb08111ac3b18953b0cebb850e40bc4d)
> * Duration 3:37:29
> * Result: ❌ **FAILED**
> * Error: `Error while executing command: if $fail_test; then exit 1; fi. Reason: exit status 1`
> * [Build Log](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator/fdb-kubernetes-operator-pr/a012dd42-c045-4081-8344-56a7a9122197.gz) terminal output (available for 30 days)
> * [Build Workspace](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator-pr/a012dd42-c045-4081-8344-56a7a9122197/workspace.zip) zip file of the working directory (available for 30 days)

```text
[Container] 2025/03/12 14:15:42.437957 Running command if $(grep -q -- "--- FAIL:" ${CODEBUILD_SRC_DIR}/logs/*.log); then echo "TESTS FAILED SEE THESE LOGS:"; echo ; grep -l -- "--- FAIL:" ${CODEBUILD_SRC_DIR}/logs/*.log; export fail_test=true; fi
TESTS FAILED SEE THESE LOGS:

/codebuild/output/src91157654/src/github.com/FoundationDB/fdb-kubernetes-operator/logs/test_operator_ha_upgrades.log
```
author_association: MEMBER
user: login: foundationdb-ci
created_at: 2025-03-18T17:55:48Z
body: 
### Result of fdb-kubernetes-operator-pr on Linux CentOS 7

* Commit ID: 6fa81b15d6696330814f09ce2e6f843635e3a7b1
* Duration 3:09:15
* Result: :white_check_mark: **SUCCEEDED**
* Error: `N/A`
* [Build Log](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator/fdb-kubernetes-operator-pr/cc46a77e-2836-4e8f-b6ee-1e51468ca7ae.gz) terminal output (available for 30 days)
* [Build Workspace](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator-pr/cc46a77e-2836-4e8f-b6ee-1e51468ca7ae/workspace.zip) zip file of the working directory (available for 30 days)

author_association: CONTRIBUTOR