Repository: FoundationDB/fdb-kubernetes-operator
Pull request #1376 (2022-10-19 15:24:35)

Make remove incompatible processes more pessimistic
# Description

*Please include a summary of the change and which issue is addressed. If this change resolves an issue, please include the issue number in the description.*

## Type of change

*Please select one of the options below.*

- Bug fix (non-breaking change which fixes an issue)
- New feature (non-breaking change which adds functionality)
- Breaking change (fix or feature that would cause existing functionality to not work as expected)
- Documentation
- Other

## Discussion

*Are there any design details that you would like to discuss further?*

## Testing

*Please describe the tests that you ran to verify your changes. Unit tests?
Manual testing?*

*Do we need to perform additional testing once this is merged, or perform in a larger testing environment?*

## Documentation

*Did you update relevant documentation within this repository?*

*If this change is adding new functionality, do we need to describe it in our user manual?*

*If this change is adding or removing subreconcilers, have we updated the core technical design doc to reflect that?*

*If this change is adding new safety checks or new potential failure modes, have we documented and how to debug potential issues?*

## Follow-up

*Are there any follow-up issues that we should pursue in the future?*

*Does this introduce new defaults that we should re-evaluate in the future?*


foundationdb-ci: 
### Result of fdb-kubernetes-operator-pr on Linux CentOS 7

* Commit ID: 29ca7e84914bf23c3e537d94e356674573028754
* Duration 4:06:04
* Result: :white_check_mark: **SUCCEEDED**
* Error: `N/A`
* [Build Logs](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator/fdb-kubernetes-operator-pr/8506e73c-dfee-4d39-9347-a6e09271b7da.gz) (available for 30 days)
* [Build Artifact](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator-pr/8506e73c-dfee-4d39-9347-a6e09271b7da/workspace.zip) (available for 30 days)

foundationdb-ci: 
### Result of fdb-kubernetes-operator-pr on Linux CentOS 7

* Commit ID: e2eb30b10e01b16e505f38d1ba104f5f1ec82417
* Duration 4:06:24
* Result: :white_check_mark: **SUCCEEDED**
* Error: `N/A`
* [Build Logs](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator/fdb-kubernetes-operator-pr/90a1386e-fa04-490a-bfd8-b2e68a6062af.gz) (available for 30 days)
* [Build Artifact](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator-pr/90a1386e-fa04-490a-bfd8-b2e68a6062af/workspace.zip) (available for 30 days)

foundationdb-ci: 
### Result of fdb-kubernetes-operator-pr on Linux CentOS 7

* Commit ID: e2eb30b10e01b16e505f38d1ba104f5f1ec82417
* Duration 2:12:17
* Result: :x: **FAILED**
* Error: `Error while executing command: if $(grep -q -- "--- FAIL:" logs/*.log); then echo "TESTS FAILED SEE THESE LOGS:"; echo ; grep -l -- "--- FAIL:" logs/*.log; exit 1; fi. Reason: exit status 1`
* [Build Logs](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator/fdb-kubernetes-operator-pr/285f860f-f30c-4df4-a753-985e5ac1ae1e.gz) (available for 30 days)
* [Build Artifact](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator-pr/285f860f-f30c-4df4-a753-985e5ac1ae1e/workspace.zip) (available for 30 days)

foundationdb-ci: 
### Result of fdb-kubernetes-operator-pr on Linux CentOS 7

* Commit ID: 0cd5e60c1796bca71cbb9bf0445b8dc1f3c83f03
* Duration 1:55:31
* Result: :white_check_mark: **SUCCEEDED**
* Error: `N/A`
* [Build Logs](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator/fdb-kubernetes-operator-pr/e312dc57-645c-4f1e-8974-38bfbe8700ef.gz) (available for 30 days)
* [Build Artifact](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator-pr/e312dc57-645c-4f1e-8974-38bfbe8700ef/workspace.zip) (available for 30 days)

foundationdb-ci: 
### Result of fdb-kubernetes-operator-pr on Linux CentOS 7

* Commit ID: 86b1615d65fc63c044231b3432d482981faa76ab
* Duration 4:06:03
* Result: :white_check_mark: **SUCCEEDED**
* Error: `N/A`
* [Build Logs](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator/fdb-kubernetes-operator-pr/d1fc1e67-b3b5-4bb1-b65d-009daf42a6fc.gz) (available for 30 days)
* [Build Artifact](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator-pr/d1fc1e67-b3b5-4bb1-b65d-009daf42a6fc/workspace.zip) (available for 30 days)

foundationdb-ci: 
### Result of fdb-kubernetes-operator-pr on Linux CentOS 7

* Commit ID: 0cd5e60c1796bca71cbb9bf0445b8dc1f3c83f03
* Duration 4:05:48
* Result: :white_check_mark: **SUCCEEDED**
* Error: `N/A`
* [Build Logs](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator/fdb-kubernetes-operator-pr/65b290ef-c5cb-4bb0-bf88-9238a4322342.gz) (available for 30 days)
* [Build Artifact](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator-pr/65b290ef-c5cb-4bb0-bf88-9238a4322342/workspace.zip) (available for 30 days)

foundationdb-ci: 
### Result of fdb-kubernetes-operator-pr on Linux CentOS 7

* Commit ID: cde4d6351c44d73384e54199233da1954824bb7f
* Duration 1:49:54
* Result: :white_check_mark: **SUCCEEDED**
* Error: `N/A`
* [Build Logs](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator/fdb-kubernetes-operator-pr/17fcc96e-b9a8-4fa0-968a-008f275e843f.gz) (available for 30 days)
* [Build Artifact](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator-pr/17fcc96e-b9a8-4fa0-968a-008f275e843f/workspace.zip) (available for 30 days)

foundationdb-ci: 
### Result of fdb-kubernetes-operator-pr on Linux CentOS 7

* Commit ID: 9a515725fbe8ec05cc64ef53b388e4154f8b140f
* Duration 1:54:10
* Result: :white_check_mark: **SUCCEEDED**
* Error: `N/A`
* [Build Logs](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator/fdb-kubernetes-operator-pr/90f9cf4e-3a32-4e5b-8357-a7f61c09bd0a.gz) (available for 30 days)
* [Build Artifact](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator-pr/90f9cf4e-3a32-4e5b-8357-a7f61c09bd0a/workspace.zip) (available for 30 days)

foundationdb-ci: 
### Result of fdb-kubernetes-operator-pr on Linux CentOS 7

* Commit ID: 9a515725fbe8ec05cc64ef53b388e4154f8b140f
* Duration 4:05:57
* Result: :white_check_mark: **SUCCEEDED**
* Error: `N/A`
* [Build Logs](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator/fdb-kubernetes-operator-pr/55520070-03cf-4cff-b639-1722a784b8b0.gz) (available for 30 days)
* [Build Artifact](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator-pr/55520070-03cf-4cff-b639-1722a784b8b0/workspace.zip) (available for 30 days)


Patch:
diff --git a/api/v1beta2/foundationdb_database_configuration.go b/api/v1beta2/foundationdb_database_configuration.go
index 84f7d482..8dcce531 100644
--- a/api/v1beta2/foundationdb_database_configuration.go
+++ b/api/v1beta2/foundationdb_database_configuration.go
@@ -579,7 +579,6 @@ func (configuration DatabaseConfiguration) AreSeparatedProxiesConfigured() bool
 // AreSeparatedProxiesConfigured(), then this function will return the
 // string "commit_proxies=%d grv_proxies=%d", otherwise just
 // "proxies=%d" using the correct counts of the configuration object.
-//
 func (configuration DatabaseConfiguration) GetProxiesString(version Version) string {
 	counts := configuration.GetRoleCountsWithDefaults(version, DesiredFaultTolerance(configuration.RedundancyMode))
 	if version.HasSeparatedProxies() && configuration.AreSeparatedProxiesConfigured() {
diff --git a/api/v1beta2/foundationdbcluster_types.go b/api/v1beta2/foundationdbcluster_types.go
index e2463915..87027d86 100644
--- a/api/v1beta2/foundationdbcluster_types.go
+++ b/api/v1beta2/foundationdbcluster_types.go
@@ -1207,6 +1207,10 @@ func (cluster *FoundationDBCluster) CheckReconciliation(log logr.Logger) (bool,
 		if len(processGroup.ProcessGroupConditions) > 0 {
 			conditions := make([]ProcessGroupConditionType, 0, len(processGroup.ProcessGroupConditions))
 			for _, condition := range processGroup.ProcessGroupConditions {
+				if condition.ProcessGroupConditionType == IncorrectCommandLine && cluster.Status.Generations.NeedsBounce == 0 {
+					logger.Info("Pending restart of fdbserver processes", "state", "NeedsBounce")
+					cluster.Status.Generations.NeedsBounce = cluster.ObjectMeta.Generation
+				}
 				conditions = append(conditions, condition.ProcessGroupConditionType)
 			}
 
diff --git a/api/v1beta2/groupversion_info.go b/api/v1beta2/groupversion_info.go
index 0d7c79e5..96fd86a4 100644
--- a/api/v1beta2/groupversion_info.go
+++ b/api/v1beta2/groupversion_info.go
@@ -15,8 +15,8 @@ limitations under the License.
 */
 
 // Package v1beta2 contains API Schema definitions for the apps v1beta2 API group
-//+kubebuilder:object:generate=true
-//+groupName=apps.foundationdb.org
+// +kubebuilder:object:generate=true
+// +groupName=apps.foundationdb.org
 package v1beta2
 
 import (
diff --git a/controllers/bounce_processes.go b/controllers/bounce_processes.go
index 0aec4f9e..e261197a 100644
--- a/controllers/bounce_processes.go
+++ b/controllers/bounce_processes.go
@@ -23,18 +23,15 @@ package controllers
 import (
 	"context"
 	"fmt"
-	"math"
 	"time"
 
+	fdbv1beta2 "github.com/FoundationDB/fdb-kubernetes-operator/api/v1beta2"
+	"github.com/FoundationDB/fdb-kubernetes-operator/internal"
 	"github.com/FoundationDB/fdb-kubernetes-operator/pkg/fdbadminclient"
 	"github.com/FoundationDB/fdb-kubernetes-operator/pkg/podmanager"
-
-	"github.com/FoundationDB/fdb-kubernetes-operator/internal"
-
+	"github.com/go-logr/logr"
 	corev1 "k8s.io/api/core/v1"
 	"k8s.io/utils/pointer"
-
-	fdbv1beta2 "github.com/FoundationDB/fdb-kubernetes-operator/api/v1beta2"
 )
 
 // bounceProcesses provides a reconciliation step for bouncing fdbserver
@@ -43,6 +40,10 @@ type bounceProcesses struct{}
 
 // reconcile runs the reconciler's work.
 func (bounceProcesses) reconcile(ctx context.Context, r *FoundationDBClusterReconciler, cluster *fdbv1beta2.FoundationDBCluster) *requeue {
+	if !pointer.BoolDeref(cluster.Spec.AutomationOptions.KillProcesses, true) {
+		return nil
+	}
+
 	logger := log.WithValues("namespace", cluster.Namespace, "cluster", cluster.Name, "reconciler", "bounceProcesses")
 	adminClient, err := r.getDatabaseClientProvider().GetAdminClient(cluster, r)
 	if err != nil {
@@ -55,15 +56,103 @@ func (bounceProcesses) reconcile(ctx context.Context, r *FoundationDBClusterReco
 		return &requeue{curError: err}
 	}
 
-	minimumUptime, addressMap, err := getMinimumUptimeAndAddressMap(cluster, status, r.EnableRecoveryState)
+	minimumUptime, addressMap, err := internal.GetMinimumUptimeAndAddressMap(cluster, status, r.EnableRecoveryState)
+	if err != nil {
+		return &requeue{curError: err}
+	}
+
+	addresses, req := getProcessesReadyForRestart(ctx, logger, r, cluster, addressMap)
+	if req != nil {
+		return req
+	}
+
+	if len(addresses) == 0 {
+		return nil
+	}
+
+	if minimumUptime < float64(cluster.GetMinimumUptimeSecondsForBounce()) {
+		r.Recorder.Event(cluster, corev1.EventTypeNormal, "NeedsBounce",
+			fmt.Sprintf("Spec require a bounce of some processes, but the cluster has only been up for %f seconds", minimumUptime))
+		cluster.Status.Generations.NeedsBounce = cluster.ObjectMeta.Generation
+		err = r.updateOrApply(ctx, cluster)
+		if err != nil {
+			logger.Error(err, "Error updating cluster status")
+		}
+
+		// Retry after we waited the minimum uptime
+		return &requeue{
+			message: "Cluster needs to stabilize before bouncing",
+			delay:   time.Second * time.Duration(cluster.GetMinimumUptimeSecondsForBounce()-int(minimumUptime)),
+		}
+	}
+
+	var lockClient fdbadminclient.LockClient
+	useLocks := cluster.ShouldUseLocks()
+	if useLocks {
+		lockClient, err = r.getLockClient(cluster)
+		if err != nil {
+			return &requeue{curError: err}
+		}
+	}
+	version, err := fdbv1beta2.ParseFdbVersion(cluster.Spec.Version)
+	if err != nil {
+		return &requeue{curError: err}
+	}
+
+	upgrading := cluster.Status.RunningVersion != cluster.Spec.Version
+
+	if useLocks && upgrading {
+		processGroupIDs := make([]string, 0, len(cluster.Status.ProcessGroups))
+		for _, processGroup := range cluster.Status.ProcessGroups {
+			processGroupIDs = append(processGroupIDs, processGroup.ProcessGroupID)
+		}
+		err = lockClient.AddPendingUpgrades(version, processGroupIDs)
+		if err != nil {
+			return &requeue{curError: err}
+		}
+	}
+
+	hasLock, err := r.takeLock(cluster, fmt.Sprintf("bouncing processes: %v", addresses))
+	if !hasLock {
+		return &requeue{curError: err}
+	}
+
+	if useLocks && upgrading {
+		var req *requeue
+		addresses, req = getAddressesForUpgrade(logger, r, status, lockClient, cluster, version)
+		if req != nil {
+			return req
+		}
+		if addresses == nil {
+			return &requeue{curError: fmt.Errorf("unknown error when getting addresses that are ready for upgrade")}
+		}
+	}
+
+	logger.Info("Bouncing processes", "addresses", addresses, "upgrading", upgrading)
+	r.Recorder.Event(cluster, corev1.EventTypeNormal, "BouncingProcesses", fmt.Sprintf("Bouncing processes: %v", addresses))
+	err = adminClient.KillProcesses(addresses)
 	if err != nil {
 		return &requeue{curError: err}
 	}
 
+	// If the cluster was upgraded we will requeue and let the update_status command set the correct version.
+	// Updating the version in this method has the drawback that we upgrade the version independent of the success
+	// of the kill command. The kill command is not reliable, which means that some kill request might not be
+	// delivered and the return value will still not contain any error.
+	if upgrading {
+		return &requeue{message: "fetch latest status after upgrade"}
+	}
+
+	return nil
+}
+
+// getProcessesReadyForRestart returns a slice of process addresses that can be restarted. If addresses are missing or not all processes
+// have the latest configuration this method will return a requeue struct with more details.
+func getProcessesReadyForRestart(ctx context.Context, logger logr.Logger, r *FoundationDBClusterReconciler, cluster *fdbv1beta2.FoundationDBCluster, addressMap map[string][]fdbv1beta2.ProcessAddress) ([]fdbv1beta2.ProcessAddress, *requeue) {
 	processesToBounce := fdbv1beta2.FilterByConditions(cluster.Status.ProcessGroups, map[fdbv1beta2.ProcessGroupConditionType]bool{
 		fdbv1beta2.IncorrectCommandLine: true,
 		fdbv1beta2.IncorrectPodSpec:     false,
-		fdbv1beta2.SidecarUnreachable:   false, // ignore all Process groups that are not reachable and therefore will not get any config map updates.
+		fdbv1beta2.SidecarUnreachable:   false, // ignore all Process groups that are not reachable and therefore will not get any ConfigMap updates.
 	}, true)
 
 	addresses := make([]fdbv1beta2.ProcessAddress, 0, len(processesToBounce))
@@ -95,10 +184,10 @@ func (bounceProcesses) reconcile(ctx context.Context, r *FoundationDBClusterReco
 		processGroupID := podmanager.GetProcessGroupIDFromProcessID(process)
 		pod, err := r.PodLifecycleManager.GetPods(ctx, r, cluster, internal.GetSinglePodListOptions(cluster, processGroupID)...)
 		if err != nil {
-			return &requeue{curError: err}
+			return nil, &requeue{curError: err}
 		}
 		if len(pod) == 0 {
-			return &requeue{message: fmt.Sprintf("No pod defined for process group ID: \"%s\"", processGroupID), delay: podSchedulingDelayDuration}
+			return nil, &requeue{message: fmt.Sprintf("No pod defined for process group ID: \"%s\"", processGroupID), delay: podSchedulingDelayDuration}
 		}
 
 		synced, err := r.updatePodDynamicConf(cluster, pod[0])
@@ -109,107 +198,19 @@ func (bounceProcesses) reconcile(ctx context.Context, r *FoundationDBClusterReco
 	}
 
 	if len(missingAddress) > 0 {
-		return &requeue{curError: fmt.Errorf("could not find address for processes: %s", missingAddress)}
+		return nil, &requeue{curError: fmt.Errorf("could not find address for processes: %s", missingAddress), delayedRequeue: true}
 	}
 
 	if !allSynced {
-		return &requeue{message: "Waiting for config map to sync to all pods"}
-	}
-
-	if len(addresses) > 0 {
-		if !pointer.BoolDeref(cluster.Spec.AutomationOptions.KillProcesses, true) {
-			r.Recorder.Event(cluster, corev1.EventTypeNormal, "NeedsBounce",
-				"Spec require a bounce of some processes, but killing processes is disabled")
-			cluster.Status.Generations.NeedsBounce = cluster.ObjectMeta.Generation
-			err = r.updateOrApply(ctx, cluster)
-			if err != nil {
-				logger.Error(err, "Error updating cluster status")
-			}
-
-			return &requeue{message: "Kills are disabled"}
-		}
-
-		if minimumUptime < float64(cluster.GetMinimumUptimeSecondsForBounce()) {
-			r.Recorder.Event(cluster, corev1.EventTypeNormal, "NeedsBounce",
-				fmt.Sprintf("Spec require a bounce of some processes, but the cluster has only been up for %f seconds", minimumUptime))
-			cluster.Status.Generations.NeedsBounce = cluster.ObjectMeta.Generation
-			err = r.updateOrApply(ctx, cluster)
-			if err != nil {
-				logger.Error(err, "Error updating cluster status")
-			}
-
-			// Retry after we waited the minimum uptime
-			return &requeue{
-				message: "Cluster needs to stabilize before bouncing",
-				delay:   time.Second * time.Duration(cluster.GetMinimumUptimeSecondsForBounce()-int(minimumUptime)),
-			}
-		}
-
-		var lockClient fdbadminclient.LockClient
-		useLocks := cluster.ShouldUseLocks()
-		if useLocks {
-			lockClient, err = r.getLockClient(cluster)
-			if err != nil {
-				return &requeue{curError: err}
-			}
-		}
-		version, err := fdbv1beta2.ParseFdbVersion(cluster.Spec.Version)
-		if err != nil {
-			return &requeue{curError: err}
-		}
-
-		upgrading := cluster.Status.RunningVersion != cluster.Spec.Version
-
-		if useLocks && upgrading {
-			processGroupIDs := make([]string, 0, len(cluster.Status.ProcessGroups))
-			for _, processGroup := range cluster.Status.ProcessGroups {
-				processGroupIDs = append(processGroupIDs, processGroup.ProcessGroupID)
-			}
-			err = lockClient.AddPendingUpgrades(version, processGroupIDs)
-			if err != nil {
-				return &requeue{curError: err}
-			}
-		}
-
-		hasLock, err := r.takeLock(cluster, fmt.Sprintf("bouncing processes: %v", addresses))
-		if !hasLock {
-			return &requeue{curError: err}
-		}
-
-		if useLocks && upgrading {
-			var req *requeue
-			addresses, req = getAddressesForUpgrade(r, status, lockClient, cluster, version)
-			if req != nil {
-				return req
-			}
-			if addresses == nil {
-				return &requeue{curError: fmt.Errorf("unknown error when getting addresses that are ready for upgrade")}
-			}
-		}
-
-		logger.Info("Bouncing processes", "addresses", addresses, "upgrading", upgrading)
-		r.Recorder.Event(cluster, corev1.EventTypeNormal, "BouncingProcesses", fmt.Sprintf("Bouncing processes: %v", addresses))
-		err = adminClient.KillProcesses(addresses)
-		if err != nil {
-			return &requeue{curError: err}
-		}
-
-		// If the cluster was upgraded we will requeue and let the update_status command set the correct version.
-		// Updating the version in this method has the drawback that we upgrade the version independent of the success
-		// of the kill command. The kill command is not reliable, which means that some kill request might not be
-		// delivered and the return value will still not contain any error.
-		if upgrading {
-			return &requeue{message: "fetch latest status after upgrade"}
-		}
+		return nil, &requeue{message: "Waiting for config map to sync to all pods", delayedRequeue: true}
 	}
 
-	return nil
+	return addresses, nil
 }
 
 // getAddressesForUpgrade checks that all processes in a cluster are ready to be
 // upgraded and returns the full list of addresses.
-func getAddressesForUpgrade(r *FoundationDBClusterReconciler, databaseStatus *fdbv1beta2.FoundationDBStatus, lockClient fdbadminclient.LockClient, cluster *fdbv1beta2.FoundationDBCluster, version fdbv1beta2.Version) ([]fdbv1beta2.ProcessAddress, *requeue) {
-	logger := log.WithValues("namespace", cluster.Namespace, "cluster", cluster.Name, "reconciler", "bounceProcesses")
+func getAddressesForUpgrade(logger logr.Logger, r *FoundationDBClusterReconciler, databaseStatus *fdbv1beta2.FoundationDBStatus, lockClient fdbadminclient.LockClient, cluster *fdbv1beta2.FoundationDBCluster, version fdbv1beta2.Version) ([]fdbv1beta2.ProcessAddress, *requeue) {
 	pendingUpgrades, err := lockClient.GetPendingUpgrades(version)
 	if err != nil {
 		return nil, &requeue{curError: err}
@@ -247,36 +248,3 @@ func getAddressesForUpgrade(r *FoundationDBClusterReconciler, databaseStatus *fd
 
 	return addresses, nil
 }
-
-// getMinimumUptimeAndAddressMap returns address map of the processes included the the foundationdb status. The minimum
-// uptime will be either secondsSinceLastRecovered if the recovery state is supported and enabled otherwise we will
-// take the minimum uptime of all processes.
-func getMinimumUptimeAndAddressMap(cluster *fdbv1beta2.FoundationDBCluster, status *fdbv1beta2.FoundationDBStatus, recoveryStateEnabled bool) (float64, map[string][]fdbv1beta2.ProcessAddress, error) {
-	runningVersion, err := fdbv1beta2.ParseFdbVersion(cluster.GetRunningVersion())
-	if err != nil {
-		return 0, nil, err
-	}
-
-	useRecoveryState := runningVersion.SupportsRecoveryState() && recoveryStateEnabled
-
-	addressMap := make(map[string][]fdbv1beta2.ProcessAddress, len(status.Cluster.Processes))
-
-	minimumUptime := math.Inf(1)
-	if useRecoveryState {
-		minimumUptime = status.Cluster.RecoveryState.SecondsSinceLastRecovered
-	}
-
-	for _, process := range status.Cluster.Processes {
-		addressMap[process.Locality[fdbv1beta2.FDBLocalityInstanceIDKey]] = append(addressMap[process.Locality[fdbv1beta2.FDBLocalityInstanceIDKey]], process.Address)
-
-		if useRecoveryState || process.Excluded {
-			continue
-		}
-
-		if process.UptimeSeconds < minimumUptime {
-			minimumUptime = process.UptimeSeconds
-		}
-	}
-
-	return minimumUptime, addressMap, nil
-}
diff --git a/controllers/remove_incompatible_processes.go b/controllers/remove_incompatible_processes.go
index 410e344d..d9762171 100644
--- a/controllers/remove_incompatible_processes.go
+++ b/controllers/remove_incompatible_processes.go
@@ -90,18 +90,32 @@ func processIncompatibleProcesses(ctx context.Context, r *FoundationDBClusterRec
 		return nil
 	}
 
-	logger.Info("incompatible connections", "incompatibleConnections", status.Cluster.IncompatibleConnections)
-	incompatibleConnections := map[string]fdbv1beta2.None{}
-	for _, incompatibleAddress := range status.Cluster.IncompatibleConnections {
-		address, err := fdbv1beta2.ParseProcessAddress(incompatibleAddress)
-		if err != nil {
-			logger.Error(err, "could not parse address in incompatible connections", "address", incompatibleAddress)
-			continue
-		}
+	// Wait until the cluster is running for the minimum uptime before looking for incompatible processes.
+	minimumUptime, _, err := internal.GetMinimumUptimeAndAddressMap(cluster, status, r.EnableRecoveryState)
+	if err != nil {
+		return err
+	}
 
-		incompatibleConnections[address.IPAddress.String()] = fdbv1beta2.None{}
+	if minimumUptime < float64(cluster.GetMinimumUptimeSecondsForBounce()) {
+		logger.V(1).Info("Skipping reconciler and waiting until cluster is for up minimum uptime")
+		return nil
+	}
+
+	// Ensure the cluster is running at fault tolerance before recreating Pods.
+	hasDesiredFaultTolerance, err := internal.HasDesiredFaultToleranceFromStatus(logger, status, cluster)
+	if err != nil {
+		logger.V(1).Info("Cluster doesn't have required fault tolerance won't delete Pods")
+		return err
 	}
 
+	if !hasDesiredFaultTolerance {
+		logger.V(1).Info("Skipping reconciler and waiting until cluster has desired fault tolerance")
+		return nil
+	}
+
+	logger.Info("incompatible connections", "incompatibleConnections", status.Cluster.IncompatibleConnections)
+	incompatibleConnections := parseIncompatibleConnections(logger, status.Cluster.IncompatibleConnections)
+
 	incompatiblePods := make([]*corev1.Pod, 0, len(incompatibleConnections))
 	for _, processGroup := range cluster.Status.ProcessGroups {
 		pod, ok := podMap[processGroup.ProcessGroupID]
@@ -111,8 +125,14 @@ func processIncompatibleProcesses(ctx context.Context, r *FoundationDBClusterRec
 			continue
 		}
 
+		if pod.DeletionTimestamp != nil {
+			logger.V(1).Info("Skipping Pod that is already marked for deletion",
+				"processGroupID", processGroup.ProcessGroupID)
+			continue
+		}
+
 		if isIncompatible(incompatibleConnections, processGroup) {
-			logger.Info("recreate Pod for process group with incompatible version", "processGroupID", processGroup.ProcessGroupID)
+			logger.Info("recreate Pod for process group with incompatible version", "processGroupID", processGroup.ProcessGroupID, "address", processGroup.Addresses)
 			incompatiblePods = append(incompatiblePods, pod)
 		}
 	}
@@ -121,6 +141,27 @@ func processIncompatibleProcesses(ctx context.Context, r *FoundationDBClusterRec
 	return r.PodLifecycleManager.UpdatePods(ctx, r, cluster, incompatiblePods, true)
 }
 
+// parseIncompatibleConnections parses the incompatible connections string slice to a map
+func parseIncompatibleConnections(logger logr.Logger, incompatibleConnections []string) map[string]fdbv1beta2.None {
+	result := make(map[string]fdbv1beta2.None)
+	for _, incompatibleAddress := range incompatibleConnections {
+		address, err := fdbv1beta2.ParseProcessAddress(incompatibleAddress)
+		if err != nil {
+			logger.Error(err, "could not parse address in incompatible connections", "address", incompatibleAddress)
+			continue
+		}
+
+		if address.Port == 0 {
+			logger.V(1).Info("Ignore incompatible connection with port 0", "address", address)
+			continue
+		}
+
+		result[address.IPAddress.String()] = fdbv1beta2.None{}
+	}
+
+	return result
+}
+
 // isIncompatible checks if the process group is in the list of incompatible connections.
 func isIncompatible(incompatibleConnections map[string]fdbv1beta2.None, processGroup *fdbv1beta2.ProcessGroupStatus) bool {
 	for _, address := range processGroup.Addresses {
diff --git a/controllers/remove_process_groups.go b/controllers/remove_process_groups.go
index 696b99a4..522d1034 100644
--- a/controllers/remove_process_groups.go
+++ b/controllers/remove_process_groups.go
@@ -71,12 +71,17 @@ func (u removeProcessGroups) reconcile(ctx context.Context, r *FoundationDBClust
 		}
 	}
 
+	status, err := adminClient.GetStatus()
+	if err != nil {
+		return &requeue{curError: err}
+	}
+
 	// We don't use the "cached" of the cluster status from the CRD to minimize the window between data loss (e.g. a node
 	// or a set of Pods is not reachable anymore). We still end up with the risk to actually query the FDB cluster and after that
 	// query the cluster gets into a degraded state.
 	// We could be smarter here and only block removals that target stateful processes by e.g. filtering those out of the
 	// processGroupsToRemove slice.
-	hasDesiredFaultTolerance, err := internal.HasDesiredFaultTolerance(logger, adminClient, cluster)
+	hasDesiredFaultTolerance, err := internal.HasDesiredFaultToleranceFromStatus(logger, status, cluster)
 	if err != nil {
 		return &requeue{curError: err}
 	}
@@ -88,11 +93,6 @@ func (u removeProcessGroups) reconcile(ctx context.Context, r *FoundationDBClust
 		}
 	}
 
-	status, err := adminClient.GetStatus()
-	if err != nil {
-		return &requeue{curError: err}
-	}
-
 	// In addition to that we should add the same logic as in the exclude step
 	// to ensure we never exclude/remove more process groups than desired.
 	zonedRemovals, lastDeletion, err := removals.GetZonedRemovals(status, processGroupsToRemove)
diff --git a/internal/fault_tolerance.go b/internal/fault_tolerance.go
index 6d3d629e..b1c76fcb 100644
--- a/internal/fault_tolerance.go
+++ b/internal/fault_tolerance.go
@@ -39,10 +39,15 @@ func HasDesiredFaultTolerance(log logr.Logger, adminClient fdbadminclient.AdminC
 		return false, err
 	}
 
+	return HasDesiredFaultToleranceFromStatus(log, status, cluster)
+}
+
+// HasDesiredFaultToleranceFromStatus checks if the cluster has the desired fault tolerance based on the provided status.
+func HasDesiredFaultToleranceFromStatus(log logr.Logger, status *fdbv1beta2.FoundationDBStatus, cluster *fdbv1beta2.FoundationDBCluster) (bool, error) {
 	if !status.Client.DatabaseStatus.Available {
-		//log.V(0).Info("Cluster is not available",
-		//	"namespace", cluster.Namespace,
-		//	"cluster", cluster.Name)
+		log.Info("Cluster is not available",
+			"namespace", cluster.Namespace,
+			"cluster", cluster.Name)
 
 		return false, nil
 	}
diff --git a/internal/foundationdb_status.go b/internal/foundationdb_status.go
index 1c4cf254..bb6ece78 100644
--- a/internal/foundationdb_status.go
+++ b/internal/foundationdb_status.go
@@ -21,26 +21,61 @@
 package internal
 
 import (
-	"github.com/FoundationDB/fdb-kubernetes-operator/api/v1beta2"
+	"math"
+
+	fdbv1beta2 "github.com/FoundationDB/fdb-kubernetes-operator/api/v1beta2"
 )
 
 // GetCoordinatorsFromStatus gets the current coordinators from the status.
 // The returning set will contain all processes by their process group ID.
-func GetCoordinatorsFromStatus(status *v1beta2.FoundationDBStatus) map[string]struct{} {
+func GetCoordinatorsFromStatus(status *fdbv1beta2.FoundationDBStatus) map[string]struct{} {
 	coordinators := make(map[string]struct{})
 
 	for _, pInfo := range status.Cluster.Processes {
 		for _, roleInfo := range pInfo.Roles {
-			if roleInfo.Role != string(v1beta2.ProcessRoleCoordinator) {
+			if roleInfo.Role != string(fdbv1beta2.ProcessRoleCoordinator) {
 				continue
 			}
 
 			// We don't have to check for duplicates here, if the process group ID is already
 			// set we just overwrite it.
-			coordinators[pInfo.Locality[v1beta2.FDBLocalityInstanceIDKey]] = struct{}{}
+			coordinators[pInfo.Locality[fdbv1beta2.FDBLocalityInstanceIDKey]] = struct{}{}
 			break
 		}
 	}
 
 	return coordinators
 }
+
+// GetMinimumUptimeAndAddressMap returns address map of the processes included the the foundationdb status. The minimum
+// uptime will be either secondsSinceLastRecovered if the recovery state is supported and enabled otherwise we will
+// take the minimum uptime of all processes.
+func GetMinimumUptimeAndAddressMap(cluster *fdbv1beta2.FoundationDBCluster, status *fdbv1beta2.FoundationDBStatus, recoveryStateEnabled bool) (float64, map[string][]fdbv1beta2.ProcessAddress, error) {
+	runningVersion, err := fdbv1beta2.ParseFdbVersion(cluster.GetRunningVersion())
+	if err != nil {
+		return 0, nil, err
+	}
+
+	useRecoveryState := runningVersion.SupportsRecoveryState() && recoveryStateEnabled
+
+	addressMap := make(map[string][]fdbv1beta2.ProcessAddress, len(status.Cluster.Processes))
+
+	minimumUptime := math.Inf(1)
+	if useRecoveryState {
+		minimumUptime = status.Cluster.RecoveryState.SecondsSinceLastRecovered
+	}
+
+	for _, process := range status.Cluster.Processes {
+		addressMap[process.Locality[fdbv1beta2.FDBLocalityInstanceIDKey]] = append(addressMap[process.Locality[fdbv1beta2.FDBLocalityInstanceIDKey]], process.Address)
+
+		if useRecoveryState || process.Excluded {
+			continue
+		}
+
+		if process.UptimeSeconds < minimumUptime {
+			minimumUptime = process.UptimeSeconds
+		}
+	}
+
+	return minimumUptime, addressMap, nil
+}


Test patch:
diff --git a/api/v1beta2/foundationdbcluster_types_test.go b/api/v1beta2/foundationdbcluster_types_test.go
index b851f87f..1a55aee5 100644
--- a/api/v1beta2/foundationdbcluster_types_test.go
+++ b/api/v1beta2/foundationdbcluster_types_test.go
@@ -3141,6 +3141,7 @@ var _ = Describe("[api] FoundationDBCluster", func() {
 				Expect(cluster.Status.Generations).To(Equal(ClusterGenerationStatus{
 					Reconciled:          1,
 					HasUnhealthyProcess: 2,
+					NeedsBounce:         2,
 				}))
 
 				cluster = createCluster()
@@ -3392,6 +3393,7 @@ var _ = Describe("[api] FoundationDBCluster", func() {
 				Expect(cluster.Status.Generations).To(Equal(ClusterGenerationStatus{
 					Reconciled:          1,
 					HasUnhealthyProcess: 2,
+					NeedsBounce:         2,
 				}))
 
 				cluster = createCluster()
diff --git a/controllers/bounce_processes_test.go b/controllers/bounce_processes_test.go
index fbaccc28..9b50f89a 100644
--- a/controllers/bounce_processes_test.go
+++ b/controllers/bounce_processes_test.go
@@ -23,7 +23,6 @@ package controllers
 import (
 	"context"
 	"fmt"
-	"net"
 	"time"
 
 	"github.com/FoundationDB/fdb-kubernetes-operator/internal"
@@ -382,111 +381,4 @@ var _ = Describe("bounceProcesses", func() {
 			})
 		})
 	})
-
-	DescribeTable("when getting the minimum uptime and the address map", func(cluster *fdbv1beta2.FoundationDBCluster, status *fdbv1beta2.FoundationDBStatus, useRecoveryState bool, expectedMinimumUptime float64, expectedAddressMap map[string][]fdbv1beta2.ProcessAddress) {
-		minimumUptime, addressMap, err := getMinimumUptimeAndAddressMap(cluster, status, useRecoveryState)
-		Expect(err).NotTo(HaveOccurred())
-		Expect(minimumUptime).To(BeNumerically("==", expectedMinimumUptime))
-		Expect(len(addressMap)).To(BeNumerically("==", len(expectedAddressMap)))
-		for key, value := range expectedAddressMap {
-			Expect(addressMap).To(HaveKeyWithValue(key, value))
-		}
-	},
-		Entry("when recovered since is not available",
-			&fdbv1beta2.FoundationDBCluster{
-				Spec: fdbv1beta2.FoundationDBClusterSpec{
-					Version: fdbv1beta2.Versions.Default.String(),
-				},
-			}, &fdbv1beta2.FoundationDBStatus{
-				Cluster: fdbv1beta2.FoundationDBStatusClusterInfo{
-					Processes: map[string]fdbv1beta2.FoundationDBStatusProcessInfo{
-						"test": {
-							Address: fdbv1beta2.ProcessAddress{
-								IPAddress: net.ParseIP("127.0.0.1"),
-							},
-							Locality: map[string]string{
-								fdbv1beta2.FDBLocalityInstanceIDKey: "test",
-							},
-							UptimeSeconds: 30.0,
-						},
-					},
-					RecoveryState: fdbv1beta2.RecoveryState{
-						SecondsSinceLastRecovered: 90.0,
-					},
-				},
-			},
-			true,
-			30.0,
-			map[string][]fdbv1beta2.ProcessAddress{
-				"test": {
-					{
-						IPAddress: net.ParseIP("127.0.0.1"),
-					},
-				},
-			}),
-		Entry("when recovered since is enabled and version supports it",
-			&fdbv1beta2.FoundationDBCluster{
-				Spec: fdbv1beta2.FoundationDBClusterSpec{
-					Version: fdbv1beta2.Versions.SupportsRecoveryState.String(),
-				},
-			}, &fdbv1beta2.FoundationDBStatus{
-				Cluster: fdbv1beta2.FoundationDBStatusClusterInfo{
-					Processes: map[string]fdbv1beta2.FoundationDBStatusProcessInfo{
-						"test": {
-							Address: fdbv1beta2.ProcessAddress{
-								IPAddress: net.ParseIP("127.0.0.1"),
-							},
-							Locality: map[string]string{
-								fdbv1beta2.FDBLocalityInstanceIDKey: "test",
-							},
-							UptimeSeconds: 30.0,
-						},
-					},
-					RecoveryState: fdbv1beta2.RecoveryState{
-						SecondsSinceLastRecovered: 90.0,
-					},
-				},
-			},
-			true,
-			90.0,
-			map[string][]fdbv1beta2.ProcessAddress{
-				"test": {
-					{
-						IPAddress: net.ParseIP("127.0.0.1"),
-					},
-				},
-			}),
-		Entry("when recovered since is disabled and version supports it",
-			&fdbv1beta2.FoundationDBCluster{
-				Spec: fdbv1beta2.FoundationDBClusterSpec{
-					Version: fdbv1beta2.Versions.SupportsRecoveryState.String(),
-				},
-			}, &fdbv1beta2.FoundationDBStatus{
-				Cluster: fdbv1beta2.FoundationDBStatusClusterInfo{
-					Processes: map[string]fdbv1beta2.FoundationDBStatusProcessInfo{
-						"test": {
-							Address: fdbv1beta2.ProcessAddress{
-								IPAddress: net.ParseIP("127.0.0.1"),
-							},
-							Locality: map[string]string{
-								fdbv1beta2.FDBLocalityInstanceIDKey: "test",
-							},
-							UptimeSeconds: 30.0,
-						},
-					},
-					RecoveryState: fdbv1beta2.RecoveryState{
-						SecondsSinceLastRecovered: 90.0,
-					},
-				},
-			},
-			false,
-			30.0,
-			map[string][]fdbv1beta2.ProcessAddress{
-				"test": {
-					{
-						IPAddress: net.ParseIP("127.0.0.1"),
-					},
-				},
-			}),
-	)
 })
diff --git a/controllers/remove_incompatible_processes_test.go b/controllers/remove_incompatible_processes_test.go
index 5c479848..f6054137 100644
--- a/controllers/remove_incompatible_processes_test.go
+++ b/controllers/remove_incompatible_processes_test.go
@@ -66,6 +66,23 @@ var _ = Describe("restart_incompatible_pods", func() {
 			true),
 	)
 
+	DescribeTable("when parsing incompatible connections", func(incompatibleConnections []string, expected map[string]fdbv1beta2.None) {
+		Expect(parseIncompatibleConnections(logr.Discard(), incompatibleConnections)).To(Equal(expected))
+	},
+		Entry("empty incompatible map",
+			[]string{},
+			map[string]fdbv1beta2.None{}),
+		Entry("nil incompatible map",
+			nil,
+			map[string]fdbv1beta2.None{}),
+		Entry("incompatible map contains one address",
+			[]string{"1.1.1.1:4500:tls"},
+			map[string]fdbv1beta2.None{"1.1.1.1": {}}),
+		Entry("incompatible map contains multiple addresses",
+			[]string{"1.1.1.1:4500:tls", "1.1.1.2:0:tls"},
+			map[string]fdbv1beta2.None{"1.1.1.1": {}}),
+	)
+
 	When("running a reconcile for the restart incompatible process reconciler", func() {
 		var cluster *fdbv1beta2.FoundationDBCluster
 		var initialCount int
@@ -94,7 +111,7 @@ var _ = Describe("restart_incompatible_pods", func() {
 			Expect(err).NotTo(HaveOccurred())
 		})
 
-		When("no incompatible processes are reported", func() {
+		When("the subreconciler is enabled", func() {
 			BeforeEach(func() {
 				clusterReconciler.EnableRestartIncompatibleProcesses = true
 				adminClient, err := newMockAdminClientUncast(cluster, k8sClient)
@@ -106,55 +123,119 @@ var _ = Describe("restart_incompatible_pods", func() {
 				}
 			})
 
-			It("should have no deletions", func() {
-				pods := &corev1.PodList{}
-				err := k8sClient.List(context.TODO(), pods, getListOptions(cluster)...)
-				Expect(err).NotTo(HaveOccurred())
-				Expect(len(pods.Items)).To(BeNumerically("==", initialCount))
+			When("no incompatible processes are reported", func() {
+				BeforeEach(func() {
+					adminClient, err := newMockAdminClientUncast(cluster, k8sClient)
+					Expect(err).NotTo(HaveOccurred())
+					adminClient.frozenStatus = &fdbv1beta2.FoundationDBStatus{
+						Cluster: fdbv1beta2.FoundationDBStatusClusterInfo{
+							IncompatibleConnections: []string{},
+						},
+					}
+				})
+
+				It("should have no deletions", func() {
+					pods := &corev1.PodList{}
+					err := k8sClient.List(context.TODO(), pods, getListOptions(cluster)...)
+					Expect(err).NotTo(HaveOccurred())
+					Expect(len(pods.Items)).To(BeNumerically("==", initialCount))
+				})
 			})
-		})
 
-		When("no matching incompatible processes are reported", func() {
-			BeforeEach(func() {
-				clusterReconciler.EnableRestartIncompatibleProcesses = true
-				adminClient, err := newMockAdminClientUncast(cluster, k8sClient)
-				Expect(err).NotTo(HaveOccurred())
-				adminClient.frozenStatus = &fdbv1beta2.FoundationDBStatus{
-					Cluster: fdbv1beta2.FoundationDBStatusClusterInfo{
-						IncompatibleConnections: []string{
-							"192.192.192.192",
+			When("no matching incompatible processes are reported", func() {
+				BeforeEach(func() {
+					adminClient, err := newMockAdminClientUncast(cluster, k8sClient)
+					Expect(err).NotTo(HaveOccurred())
+					adminClient.frozenStatus = &fdbv1beta2.FoundationDBStatus{
+						Cluster: fdbv1beta2.FoundationDBStatusClusterInfo{
+							IncompatibleConnections: []string{
+								"192.192.192.192:4500:tls",
+							},
 						},
-					},
-				}
-			})
+					}
+				})
 
-			It("should have no deletions", func() {
-				pods := &corev1.PodList{}
-				err := k8sClient.List(context.TODO(), pods, getListOptions(cluster)...)
-				Expect(err).NotTo(HaveOccurred())
-				Expect(len(pods.Items)).To(BeNumerically("==", initialCount))
+				It("should have no deletions", func() {
+					pods := &corev1.PodList{}
+					err := k8sClient.List(context.TODO(), pods, getListOptions(cluster)...)
+					Expect(err).NotTo(HaveOccurred())
+					Expect(len(pods.Items)).To(BeNumerically("==", initialCount))
+				})
 			})
-		})
 
-		When("matching incompatible processes are reported and the subreconciler is enabled", func() {
-			BeforeEach(func() {
-				clusterReconciler.EnableRestartIncompatibleProcesses = true
-				adminClient, err := newMockAdminClientUncast(cluster, k8sClient)
-				Expect(err).NotTo(HaveOccurred())
-				adminClient.frozenStatus = &fdbv1beta2.FoundationDBStatus{
-					Cluster: fdbv1beta2.FoundationDBStatusClusterInfo{
-						IncompatibleConnections: []string{
-							cluster.Status.ProcessGroups[0].Addresses[0],
+			When("matching incompatible processes are reported", func() {
+				BeforeEach(func() {
+					adminClient, err := newMockAdminClientUncast(cluster, k8sClient)
+					Expect(err).NotTo(HaveOccurred())
+					adminClient.frozenStatus = &fdbv1beta2.FoundationDBStatus{
+						Client: fdbv1beta2.FoundationDBStatusLocalClientInfo{
+							DatabaseStatus: fdbv1beta2.FoundationDBStatusClientDBStatus{
+								Available: true,
+							},
 						},
-					},
-				}
-			})
+						Cluster: fdbv1beta2.FoundationDBStatusClusterInfo{
+							FaultTolerance: fdbv1beta2.FaultTolerance{
+								MaxZoneFailuresWithoutLosingAvailability: 2,
+								MaxZoneFailuresWithoutLosingData:         2,
+							},
+							IncompatibleConnections: []string{
+								cluster.Status.ProcessGroups[0].Addresses[0] + ":4500:tls",
+							},
+						},
+					}
+				})
 
-			It("should have one deletion", func() {
-				pods := &corev1.PodList{}
-				err := k8sClient.List(context.TODO(), pods, getListOptions(cluster)...)
-				Expect(err).NotTo(HaveOccurred())
-				Expect(len(pods.Items)).To(BeNumerically("==", initialCount-1))
+				It("should have one deletion", func() {
+					pods := &corev1.PodList{}
+					err := k8sClient.List(context.TODO(), pods, getListOptions(cluster)...)
+					Expect(err).NotTo(HaveOccurred())
+					Expect(len(pods.Items)).To(BeNumerically("==", initialCount-1))
+				})
+
+				When("the database is unavailable", func() {
+					BeforeEach(func() {
+						adminClient, err := newMockAdminClientUncast(cluster, k8sClient)
+						Expect(err).NotTo(HaveOccurred())
+						adminClient.frozenStatus.Client.DatabaseStatus.Available = false
+					})
+
+					It("should have no deletions", func() {
+						pods := &corev1.PodList{}
+						err := k8sClient.List(context.TODO(), pods, getListOptions(cluster)...)
+						Expect(err).NotTo(HaveOccurred())
+						Expect(len(pods.Items)).To(BeNumerically("==", initialCount))
+					})
+				})
+
+				When("matching incompatible processes are reported but port is zero", func() {
+					BeforeEach(func() {
+						adminClient, err := newMockAdminClientUncast(cluster, k8sClient)
+						Expect(err).NotTo(HaveOccurred())
+						adminClient.frozenStatus.Cluster.IncompatibleConnections[0] = cluster.Status.ProcessGroups[0].Addresses[0] + ":0:tls"
+					})
+
+					It("should have no deletions", func() {
+						pods := &corev1.PodList{}
+						err := k8sClient.List(context.TODO(), pods, getListOptions(cluster)...)
+						Expect(err).NotTo(HaveOccurred())
+						Expect(len(pods.Items)).To(BeNumerically("==", initialCount))
+					})
+				})
+
+				When("the cluster is currently upgraded", func() {
+					BeforeEach(func() {
+						cluster.Spec.Version = fdbv1beta2.Versions.NextMajorVersion.String()
+						err := k8sClient.Update(context.Background(), cluster)
+						Expect(err).NotTo(HaveOccurred())
+					})
+
+					It("should have no deletions", func() {
+						pods := &corev1.PodList{}
+						err := k8sClient.List(context.TODO(), pods, getListOptions(cluster)...)
+						Expect(err).NotTo(HaveOccurred())
+						Expect(len(pods.Items)).To(BeNumerically("==", initialCount))
+					})
+				})
 			})
 		})
 
@@ -164,34 +245,18 @@ var _ = Describe("restart_incompatible_pods", func() {
 				adminClient, err := newMockAdminClientUncast(cluster, k8sClient)
 				Expect(err).NotTo(HaveOccurred())
 				adminClient.frozenStatus = &fdbv1beta2.FoundationDBStatus{
-					Cluster: fdbv1beta2.FoundationDBStatusClusterInfo{
-						IncompatibleConnections: []string{
-							cluster.Status.ProcessGroups[0].Addresses[0],
+					Client: fdbv1beta2.FoundationDBStatusLocalClientInfo{
+						DatabaseStatus: fdbv1beta2.FoundationDBStatusClientDBStatus{
+							Available: true,
 						},
 					},
-				}
-			})
-
-			It("should have no deletions", func() {
-				pods := &corev1.PodList{}
-				err := k8sClient.List(context.TODO(), pods, getListOptions(cluster)...)
-				Expect(err).NotTo(HaveOccurred())
-				Expect(len(pods.Items)).To(BeNumerically("==", initialCount))
-			})
-		})
-
-		When("matching incompatible processes are reported and the cluster is currently upgraded", func() {
-			BeforeEach(func() {
-				cluster.Spec.Version = fdbv1beta2.Versions.NextMajorVersion.String()
-				err := k8sClient.Update(context.Background(), cluster)
-				Expect(err).NotTo(HaveOccurred())
-				clusterReconciler.EnableRestartIncompatibleProcesses = false
-				adminClient, err := newMockAdminClientUncast(cluster, k8sClient)
-				Expect(err).NotTo(HaveOccurred())
-				adminClient.frozenStatus = &fdbv1beta2.FoundationDBStatus{
 					Cluster: fdbv1beta2.FoundationDBStatusClusterInfo{
+						FaultTolerance: fdbv1beta2.FaultTolerance{
+							MaxZoneFailuresWithoutLosingAvailability: 2,
+							MaxZoneFailuresWithoutLosingData:         2,
+						},
 						IncompatibleConnections: []string{
-							cluster.Status.ProcessGroups[0].Addresses[0],
+							cluster.Status.ProcessGroups[0].Addresses[0] + ":4500:tls",
 						},
 					},
 				}
diff --git a/internal/fault_tolerance_test.go b/internal/fault_tolerance_test.go
index 3a5421fe..22574bae 100644
--- a/internal/fault_tolerance_test.go
+++ b/internal/fault_tolerance_test.go
@@ -21,8 +21,11 @@
 package internal
 
 import (
+	fdbv1beta2 "github.com/FoundationDB/fdb-kubernetes-operator/api/v1beta2"
+	"github.com/go-logr/logr"
 	. "github.com/onsi/ginkgo/v2"
 	. "github.com/onsi/gomega"
+	logf "sigs.k8s.io/controller-runtime/pkg/log"
 )
 
 var _ = Describe("fault_tolerance", func() {
@@ -64,4 +67,104 @@ var _ = Describe("fault_tolerance", func() {
 				}),
 		)
 	})
+
+	When("checking if the cluster has the desired fault tolerance from the status", func() {
+		log := logr.New(logf.NewDelegatingLogSink(logf.NullLogSink{}))
+
+		DescribeTable("should return if the cluster has the desired fault tolerance",
+			func(status *fdbv1beta2.FoundationDBStatus, cluster *fdbv1beta2.FoundationDBCluster, expected bool) {
+				Expect(HasDesiredFaultToleranceFromStatus(log, status, cluster)).To(Equal(expected))
+			},
+			Entry("cluster is fully replicated",
+				&fdbv1beta2.FoundationDBStatus{
+					Client: fdbv1beta2.FoundationDBStatusLocalClientInfo{
+						DatabaseStatus: fdbv1beta2.FoundationDBStatusClientDBStatus{
+							Available: true,
+						},
+					},
+					Cluster: fdbv1beta2.FoundationDBStatusClusterInfo{
+						FaultTolerance: fdbv1beta2.FaultTolerance{
+							MaxZoneFailuresWithoutLosingData:         2,
+							MaxZoneFailuresWithoutLosingAvailability: 2,
+						},
+					},
+				},
+				&fdbv1beta2.FoundationDBCluster{
+					Spec: fdbv1beta2.FoundationDBClusterSpec{
+						DatabaseConfiguration: fdbv1beta2.DatabaseConfiguration{
+							RedundancyMode: fdbv1beta2.RedundancyModeTriple,
+						},
+					},
+				},
+				true),
+			Entry("database is unavailable",
+				&fdbv1beta2.FoundationDBStatus{
+					Client: fdbv1beta2.FoundationDBStatusLocalClientInfo{
+						DatabaseStatus: fdbv1beta2.FoundationDBStatusClientDBStatus{
+							Available: false,
+						},
+					},
+					Cluster: fdbv1beta2.FoundationDBStatusClusterInfo{
+						FaultTolerance: fdbv1beta2.FaultTolerance{
+							MaxZoneFailuresWithoutLosingData:         2,
+							MaxZoneFailuresWithoutLosingAvailability: 2,
+						},
+					},
+				},
+				&fdbv1beta2.FoundationDBCluster{
+					Spec: fdbv1beta2.FoundationDBClusterSpec{
+						DatabaseConfiguration: fdbv1beta2.DatabaseConfiguration{
+							RedundancyMode: fdbv1beta2.RedundancyModeTriple,
+						},
+					},
+				},
+				false),
+			Entry("data is degraded",
+				&fdbv1beta2.FoundationDBStatus{
+					Client: fdbv1beta2.FoundationDBStatusLocalClientInfo{
+						DatabaseStatus: fdbv1beta2.FoundationDBStatusClientDBStatus{
+							Available: true,
+						},
+					},
+					Cluster: fdbv1beta2.FoundationDBStatusClusterInfo{
+						FaultTolerance: fdbv1beta2.FaultTolerance{
+							MaxZoneFailuresWithoutLosingData:         1,
+							MaxZoneFailuresWithoutLosingAvailability: 2,
+						},
+					},
+				},
+				&fdbv1beta2.FoundationDBCluster{
+					Spec: fdbv1beta2.FoundationDBClusterSpec{
+						DatabaseConfiguration: fdbv1beta2.DatabaseConfiguration{
+							RedundancyMode: fdbv1beta2.RedundancyModeTriple,
+						},
+					},
+				},
+				false),
+			Entry("availability is degraded",
+				&fdbv1beta2.FoundationDBStatus{
+					Client: fdbv1beta2.FoundationDBStatusLocalClientInfo{
+						DatabaseStatus: fdbv1beta2.FoundationDBStatusClientDBStatus{
+							Available: true,
+						},
+					},
+					Cluster: fdbv1beta2.FoundationDBStatusClusterInfo{
+						FaultTolerance: fdbv1beta2.FaultTolerance{
+							MaxZoneFailuresWithoutLosingData:         2,
+							MaxZoneFailuresWithoutLosingAvailability: 1,
+						},
+					},
+				},
+				&fdbv1beta2.FoundationDBCluster{
+					Spec: fdbv1beta2.FoundationDBClusterSpec{
+						DatabaseConfiguration: fdbv1beta2.DatabaseConfiguration{
+							RedundancyMode: fdbv1beta2.RedundancyModeTriple,
+						},
+					},
+				},
+				false),
+		)
+	})
+
+	// HasDesiredFaultToleranceFromStatus
 })
diff --git a/internal/foundationdb_status_test.go b/internal/foundationdb_status_test.go
index 41b15fb7..eb6aced2 100644
--- a/internal/foundationdb_status_test.go
+++ b/internal/foundationdb_status_test.go
@@ -21,7 +21,9 @@
 package internal
 
 import (
-	"github.com/FoundationDB/fdb-kubernetes-operator/api/v1beta2"
+	"net"
+
+	fdbv1beta2 "github.com/FoundationDB/fdb-kubernetes-operator/api/v1beta2"
 	. "github.com/onsi/ginkgo/v2"
 	. "github.com/onsi/gomega"
 )
@@ -29,7 +31,7 @@ import (
 var _ = Describe("Internal FoundationDBStatus", func() {
 	When("parsing the status for coordinators", func() {
 		type testCase struct {
-			status   *v1beta2.FoundationDBStatus
+			status   *fdbv1beta2.FoundationDBStatus
 			expected map[string]struct{}
 		}
 
@@ -40,19 +42,19 @@ var _ = Describe("Internal FoundationDBStatus", func() {
 			},
 			Entry("no coordinators",
 				testCase{
-					status:   &v1beta2.FoundationDBStatus{},
+					status:   &fdbv1beta2.FoundationDBStatus{},
 					expected: map[string]struct{}{},
 				}),
 			Entry("single coordinators",
 				testCase{
-					status: &v1beta2.FoundationDBStatus{
-						Cluster: v1beta2.FoundationDBStatusClusterInfo{
-							Processes: map[string]v1beta2.FoundationDBStatusProcessInfo{
+					status: &fdbv1beta2.FoundationDBStatus{
+						Cluster: fdbv1beta2.FoundationDBStatusClusterInfo{
+							Processes: map[string]fdbv1beta2.FoundationDBStatusProcessInfo{
 								"foo": {
 									Locality: map[string]string{
-										v1beta2.FDBLocalityInstanceIDKey: "foo",
+										fdbv1beta2.FDBLocalityInstanceIDKey: "foo",
 									},
-									Roles: []v1beta2.FoundationDBStatusProcessRoleInfo{
+									Roles: []fdbv1beta2.FoundationDBStatusProcessRoleInfo{
 										{
 											Role: "coordinator",
 										},
@@ -60,7 +62,7 @@ var _ = Describe("Internal FoundationDBStatus", func() {
 								},
 								"bar": {
 									Locality: map[string]string{
-										v1beta2.FDBLocalityInstanceIDKey: "bar",
+										fdbv1beta2.FDBLocalityInstanceIDKey: "bar",
 									},
 								},
 							},
@@ -72,14 +74,14 @@ var _ = Describe("Internal FoundationDBStatus", func() {
 				}),
 			Entry("multiple coordinators",
 				testCase{
-					status: &v1beta2.FoundationDBStatus{
-						Cluster: v1beta2.FoundationDBStatusClusterInfo{
-							Processes: map[string]v1beta2.FoundationDBStatusProcessInfo{
+					status: &fdbv1beta2.FoundationDBStatus{
+						Cluster: fdbv1beta2.FoundationDBStatusClusterInfo{
+							Processes: map[string]fdbv1beta2.FoundationDBStatusProcessInfo{
 								"foo": {
 									Locality: map[string]string{
-										v1beta2.FDBLocalityInstanceIDKey: "foo",
+										fdbv1beta2.FDBLocalityInstanceIDKey: "foo",
 									},
-									Roles: []v1beta2.FoundationDBStatusProcessRoleInfo{
+									Roles: []fdbv1beta2.FoundationDBStatusProcessRoleInfo{
 										{
 											Role: "coordinator",
 										},
@@ -87,9 +89,9 @@ var _ = Describe("Internal FoundationDBStatus", func() {
 								},
 								"bar": {
 									Locality: map[string]string{
-										v1beta2.FDBLocalityInstanceIDKey: "bar",
+										fdbv1beta2.FDBLocalityInstanceIDKey: "bar",
 									},
-									Roles: []v1beta2.FoundationDBStatusProcessRoleInfo{
+									Roles: []fdbv1beta2.FoundationDBStatusProcessRoleInfo{
 										{
 											Role: "coordinator",
 										},
@@ -105,4 +107,111 @@ var _ = Describe("Internal FoundationDBStatus", func() {
 				}),
 		)
 	})
+
+	DescribeTable("when getting the minimum uptime and the address map", func(cluster *fdbv1beta2.FoundationDBCluster, status *fdbv1beta2.FoundationDBStatus, useRecoveryState bool, expectedMinimumUptime float64, expectedAddressMap map[string][]fdbv1beta2.ProcessAddress) {
+		minimumUptime, addressMap, err := GetMinimumUptimeAndAddressMap(cluster, status, useRecoveryState)
+		Expect(err).NotTo(HaveOccurred())
+		Expect(minimumUptime).To(BeNumerically("==", expectedMinimumUptime))
+		Expect(len(addressMap)).To(BeNumerically("==", len(expectedAddressMap)))
+		for key, value := range expectedAddressMap {
+			Expect(addressMap).To(HaveKeyWithValue(key, value))
+		}
+	},
+		Entry("when recovered since is not available",
+			&fdbv1beta2.FoundationDBCluster{
+				Spec: fdbv1beta2.FoundationDBClusterSpec{
+					Version: fdbv1beta2.Versions.Default.String(),
+				},
+			}, &fdbv1beta2.FoundationDBStatus{
+				Cluster: fdbv1beta2.FoundationDBStatusClusterInfo{
+					Processes: map[string]fdbv1beta2.FoundationDBStatusProcessInfo{
+						"test": {
+							Address: fdbv1beta2.ProcessAddress{
+								IPAddress: net.ParseIP("127.0.0.1"),
+							},
+							Locality: map[string]string{
+								fdbv1beta2.FDBLocalityInstanceIDKey: "test",
+							},
+							UptimeSeconds: 30.0,
+						},
+					},
+					RecoveryState: fdbv1beta2.RecoveryState{
+						SecondsSinceLastRecovered: 90.0,
+					},
+				},
+			},
+			true,
+			30.0,
+			map[string][]fdbv1beta2.ProcessAddress{
+				"test": {
+					{
+						IPAddress: net.ParseIP("127.0.0.1"),
+					},
+				},
+			}),
+		Entry("when recovered since is enabled and version supports it",
+			&fdbv1beta2.FoundationDBCluster{
+				Spec: fdbv1beta2.FoundationDBClusterSpec{
+					Version: fdbv1beta2.Versions.SupportsRecoveryState.String(),
+				},
+			}, &fdbv1beta2.FoundationDBStatus{
+				Cluster: fdbv1beta2.FoundationDBStatusClusterInfo{
+					Processes: map[string]fdbv1beta2.FoundationDBStatusProcessInfo{
+						"test": {
+							Address: fdbv1beta2.ProcessAddress{
+								IPAddress: net.ParseIP("127.0.0.1"),
+							},
+							Locality: map[string]string{
+								fdbv1beta2.FDBLocalityInstanceIDKey: "test",
+							},
+							UptimeSeconds: 30.0,
+						},
+					},
+					RecoveryState: fdbv1beta2.RecoveryState{
+						SecondsSinceLastRecovered: 90.0,
+					},
+				},
+			},
+			true,
+			90.0,
+			map[string][]fdbv1beta2.ProcessAddress{
+				"test": {
+					{
+						IPAddress: net.ParseIP("127.0.0.1"),
+					},
+				},
+			}),
+		Entry("when recovered since is disabled and version supports it",
+			&fdbv1beta2.FoundationDBCluster{
+				Spec: fdbv1beta2.FoundationDBClusterSpec{
+					Version: fdbv1beta2.Versions.SupportsRecoveryState.String(),
+				},
+			}, &fdbv1beta2.FoundationDBStatus{
+				Cluster: fdbv1beta2.FoundationDBStatusClusterInfo{
+					Processes: map[string]fdbv1beta2.FoundationDBStatusProcessInfo{
+						"test": {
+							Address: fdbv1beta2.ProcessAddress{
+								IPAddress: net.ParseIP("127.0.0.1"),
+							},
+							Locality: map[string]string{
+								fdbv1beta2.FDBLocalityInstanceIDKey: "test",
+							},
+							UptimeSeconds: 30.0,
+						},
+					},
+					RecoveryState: fdbv1beta2.RecoveryState{
+						SecondsSinceLastRecovered: 90.0,
+					},
+				},
+			},
+			false,
+			30.0,
+			map[string][]fdbv1beta2.ProcessAddress{
+				"test": {
+					{
+						IPAddress: net.ParseIP("127.0.0.1"),
+					},
+				},
+			}),
+	)
 })