Repository: FoundationDB/fdb-kubernetes-operator
Pull request #1415 (2022-10-25 16:50:16)

Initial refactoring of the coordinator selection code
# Description

Initial approach to refactor some code of the coordinator selection code and add some additional testing around this code.

Fixes: https://github.com/FoundationDB/fdb-kubernetes-operator/issues/1413

## Type of change

*Please select one of the options below.*

- Other (refactoring)

## Discussion

*Are there any design details that you would like to discuss further?*

## Testing

Unit testing.

## Documentation

-

## Follow-up

-


foundationdb-ci: 
### Result of fdb-kubernetes-operator-pr on Linux CentOS 7

* Commit ID: e9c156419b89701ceada0bee25e8da312d96e63b
* Duration 4:06:08
* Result: :white_check_mark: **SUCCEEDED**
* Error: `N/A`
* [Build Logs](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator/fdb-kubernetes-operator-pr/dea017f4-54d0-47da-b4dd-ee3bfebbf9e7.gz) (available for 30 days)
* [Build Artifact](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator-pr/dea017f4-54d0-47da-b4dd-ee3bfebbf9e7/workspace.zip) (available for 30 days)

foundationdb-ci: 
### Result of fdb-kubernetes-operator-pr on Linux CentOS 7

* Commit ID: b606f18ad6a9c11ba6cfbeb27bcdc407d9ee5083
* Duration 4:08:35
* Result: :x: **FAILED**
* Error: `Error while executing command: if $(grep -q -- "--- FAIL:" logs/*.log); then echo "TESTS FAILED SEE THESE LOGS:"; echo ; grep -l -- "--- FAIL:" logs/*.log; exit 1; fi. Reason: exit status 1`
* [Build Logs](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator/fdb-kubernetes-operator-pr/b1b267f5-7bf4-47d5-8fce-3a53ede340a5.gz) (available for 30 days)
* [Build Artifact](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator-pr/b1b267f5-7bf4-47d5-8fce-3a53ede340a5/workspace.zip) (available for 30 days)

foundationdb-ci: 
### Result of fdb-kubernetes-operator-pr on Linux CentOS 7

* Commit ID: f3b7422f4e2489e3d0b2302e92cd80ed76758a8e
* Duration 4:08:33
* Result: :x: **FAILED**
* Error: `Error while executing command: if $(grep -q -- "--- FAIL:" logs/*.log); then echo "TESTS FAILED SEE THESE LOGS:"; echo ; grep -l -- "--- FAIL:" logs/*.log; exit 1; fi. Reason: exit status 1`
* [Build Logs](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator/fdb-kubernetes-operator-pr/2add3ebd-3595-45b8-9cb5-2f970a717bbe.gz) (available for 30 days)
* [Build Artifact](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator-pr/2add3ebd-3595-45b8-9cb5-2f970a717bbe/workspace.zip) (available for 30 days)

foundationdb-ci: 
### Result of fdb-kubernetes-operator-pr on Linux CentOS 7

* Commit ID: f19ba9ef5f9a81987abb6c307f1e58f798211727
* Duration 4:08:31
* Result: :x: **FAILED**
* Error: `Error while executing command: if $(grep -q -- "--- FAIL:" logs/*.log); then echo "TESTS FAILED SEE THESE LOGS:"; echo ; grep -l -- "--- FAIL:" logs/*.log; exit 1; fi. Reason: exit status 1`
* [Build Logs](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator/fdb-kubernetes-operator-pr/e4aa0598-8d33-430b-b539-2d1af8061a6c.gz) (available for 30 days)
* [Build Artifact](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator-pr/e4aa0598-8d33-430b-b539-2d1af8061a6c/workspace.zip) (available for 30 days)

foundationdb-ci: 
### Result of fdb-kubernetes-operator-pr on Linux CentOS 7

* Commit ID: 5f28716a7c7d3a4880dbf6dd9bedb89c2fab5216
* Duration 1:58:57
* Result: :x: **FAILED**
* Error: `Error while executing command: if $(grep -q -- "--- FAIL:" logs/*.log); then echo "TESTS FAILED SEE THESE LOGS:"; echo ; grep -l -- "--- FAIL:" logs/*.log; exit 1; fi. Reason: exit status 1`
* [Build Logs](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator/fdb-kubernetes-operator-pr/438ee7b5-05ac-417d-9107-b2da7a804664.gz) (available for 30 days)
* [Build Artifact](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator-pr/438ee7b5-05ac-417d-9107-b2da7a804664/workspace.zip) (available for 30 days)

foundationdb-ci: 
### Result of fdb-kubernetes-operator-pr on Linux CentOS 7

* Commit ID: bb655097f2f45df5cf79f6990a5d5726f61580d8
* Duration 1:56:40
* Result: :x: **FAILED**
* Error: `Error while executing command: if $(grep -q -- "--- FAIL:" logs/*.log); then echo "TESTS FAILED SEE THESE LOGS:"; echo ; grep -l -- "--- FAIL:" logs/*.log; exit 1; fi. Reason: exit status 1`
* [Build Logs](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator/fdb-kubernetes-operator-pr/816b1bdf-addf-4f68-a97a-8ceee456f6d2.gz) (available for 30 days)
* [Build Artifact](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator-pr/816b1bdf-addf-4f68-a97a-8ceee456f6d2/workspace.zip) (available for 30 days)

foundationdb-ci: 
### Result of fdb-kubernetes-operator-pr on Linux CentOS 7

* Commit ID: 193a2e004db5b327b0e2c7fcdabf0152e9c5e5d8
* Duration 0:04:20
* Result: :x: **FAILED**
* Error: `Error while executing command: docker build -t ${OPERATOR_IMAGE} -f Dockerfile .. Reason: exit status 2`
* [Build Logs](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator/fdb-kubernetes-operator-pr/a071bb86-5f67-4602-9a2d-7adb516a079c.gz) (available for 30 days)
* [Build Artifact](https://d1e0l78xbkh3xa.cloudfront.net) (available for 30 days)

foundationdb-ci: 
### Result of fdb-kubernetes-operator-pr on Linux CentOS 7

* Commit ID: 7157d09320a1e6e29affc4c11718cea28ce182fb
* Duration 4:06:15
* Result: :white_check_mark: **SUCCEEDED**
* Error: `N/A`
* [Build Logs](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator/fdb-kubernetes-operator-pr/d034cfa8-a503-4dc0-9b86-725f93968aba.gz) (available for 30 days)
* [Build Artifact](https://d1e0l78xbkh3xa.cloudfront.net/fdb-kubernetes-operator-pr/d034cfa8-a503-4dc0-9b86-725f93968aba/workspace.zip) (available for 30 days)


Patch:
diff --git a/api/v1beta2/foundationdb_status.go b/api/v1beta2/foundationdb_status.go
index 413334be..7bc83756 100644
--- a/api/v1beta2/foundationdb_status.go
+++ b/api/v1beta2/foundationdb_status.go
@@ -99,6 +99,9 @@ type FoundationDBStatusClusterInfo struct {
 
 	// RecoveryState represents the recovery state.
 	RecoveryState RecoveryState `json:"recovery_state,omitempty"`
+
+	// ConnectionString represents the connection string in the cluster status json output.
+	ConnectionString string `json:"connection_string,omitempty"`
 }
 
 // FaultTolerance provides information about the fault tolerance status
diff --git a/controllers/admin_client_mock.go b/controllers/admin_client_mock.go
index f647a7d4..354d06f0 100644
--- a/controllers/admin_client_mock.go
+++ b/controllers/admin_client_mock.go
@@ -123,6 +123,7 @@ func (client *mockAdminClient) GetStatus() (*fdbv1beta2.FoundationDBStatus, erro
 	var coordinatorAddresses []string
 	if strings.Contains(client.Cluster.Status.ConnectionString, "@") {
 		coordinatorAddresses = strings.Split(strings.Split(client.Cluster.Status.ConnectionString, "@")[1], ",")
+		status.Cluster.ConnectionString = client.Cluster.Status.ConnectionString
 	} else {
 		coordinatorAddresses = []string{}
 	}
diff --git a/controllers/change_coordinators.go b/controllers/change_coordinators.go
index e3ca5f15..850535be 100644
--- a/controllers/change_coordinators.go
+++ b/controllers/change_coordinators.go
@@ -23,6 +23,8 @@ package controllers
 import (
 	"context"
 	"fmt"
+	"github.com/FoundationDB/fdb-kubernetes-operator/internal/locality"
+	"github.com/go-logr/logr"
 
 	corev1 "k8s.io/api/core/v1"
 
@@ -47,15 +49,15 @@ func (c changeCoordinators) reconcile(ctx context.Context, r *FoundationDBCluste
 	}
 	defer adminClient.Close()
 
-	connectionString, err := adminClient.GetConnectionString()
+	status, err := adminClient.GetStatus()
 	if err != nil {
 		return &requeue{curError: err}
 	}
 
-	if connectionString != cluster.Status.ConnectionString {
+	if status.Cluster.ConnectionString != cluster.Status.ConnectionString {
 		logger.Info("Updating out-of-date connection string")
-		r.Recorder.Event(cluster, corev1.EventTypeNormal, "UpdatingConnectionString", fmt.Sprintf("Setting connection string to %s", connectionString))
-		cluster.Status.ConnectionString = connectionString
+		r.Recorder.Event(cluster, corev1.EventTypeNormal, "UpdatingConnectionString", fmt.Sprintf("Setting connection string to %s", status.Cluster.ConnectionString))
+		cluster.Status.ConnectionString = status.Cluster.ConnectionString
 		err = r.updateOrApply(ctx, cluster)
 
 		if err != nil {
@@ -63,17 +65,12 @@ func (c changeCoordinators) reconcile(ctx context.Context, r *FoundationDBCluste
 		}
 	}
 
-	status, err := adminClient.GetStatus()
-	if err != nil {
-		return &requeue{curError: err}
-	}
-
 	coordinatorStatus := make(map[string]bool, len(status.Client.Coordinators.Coordinators))
 	for _, coordinator := range status.Client.Coordinators.Coordinators {
 		coordinatorStatus[coordinator.Address.String()] = false
 	}
 
-	hasValidCoordinators, allAddressesValid, err := checkCoordinatorValidity(cluster, status, coordinatorStatus)
+	hasValidCoordinators, allAddressesValid, err := locality.CheckCoordinatorValidity(logger, cluster, status, coordinatorStatus)
 	if err != nil {
 		return &requeue{curError: err}
 	}
@@ -96,7 +93,7 @@ func (c changeCoordinators) reconcile(ctx context.Context, r *FoundationDBCluste
 	logger.Info("Changing coordinators")
 	r.Recorder.Event(cluster, corev1.EventTypeNormal, "ChangingCoordinators", "Choosing new coordinators")
 
-	coordinators, err := selectCoordinators(cluster, status)
+	coordinators, err := selectCoordinators(logger, cluster, status)
 	if err != nil {
 		return &requeue{curError: err}
 	}
@@ -107,7 +104,7 @@ func (c changeCoordinators) reconcile(ctx context.Context, r *FoundationDBCluste
 	}
 
 	logger.Info("Final coordinators candidates", "coordinators", coordinatorAddresses)
-	connectionString, err = adminClient.ChangeCoordinators(coordinatorAddresses)
+	connectionString, err := adminClient.ChangeCoordinators(coordinatorAddresses)
 	if err != nil {
 		return &requeue{curError: err}
 	}
@@ -120,9 +117,10 @@ func (c changeCoordinators) reconcile(ctx context.Context, r *FoundationDBCluste
 	return nil
 }
 
+// TODO move them into separate package?
 // selectCandidates is a helper for Reconcile that picks non-excluded, not-being-removed class-matching process groups.
-func selectCandidates(cluster *fdbv1beta2.FoundationDBCluster, status *fdbv1beta2.FoundationDBStatus) ([]localityInfo, error) {
-	candidates := make([]localityInfo, 0, len(status.Cluster.Processes))
+func selectCandidates(cluster *fdbv1beta2.FoundationDBCluster, status *fdbv1beta2.FoundationDBStatus) ([]locality.Info, error) {
+	candidates := make([]locality.Info, 0, len(status.Cluster.Processes))
 	for _, process := range status.Cluster.Processes {
 		if process.Excluded {
 			continue
@@ -136,29 +134,28 @@ func selectCandidates(cluster *fdbv1beta2.FoundationDBCluster, status *fdbv1beta
 			continue
 		}
 
-		locality, err := localityInfoForProcess(process, cluster.Spec.MainContainer.EnableTLS)
+		currentLocality, err := locality.InfoForProcess(process, cluster.Spec.MainContainer.EnableTLS)
 		if err != nil {
 			return candidates, err
 		}
 
-		candidates = append(candidates, locality)
+		candidates = append(candidates, currentLocality)
 	}
 
 	return candidates, nil
 }
 
-func selectCoordinators(cluster *fdbv1beta2.FoundationDBCluster, status *fdbv1beta2.FoundationDBStatus) ([]localityInfo, error) {
-	logger := log.WithValues("namespace", cluster.Namespace, "cluster", cluster.Name, "reconciler", "changeCoordinators")
+func selectCoordinators(logger logr.Logger, cluster *fdbv1beta2.FoundationDBCluster, status *fdbv1beta2.FoundationDBStatus) ([]locality.Info, error) {
 	var err error
 	coordinatorCount := cluster.DesiredCoordinatorCount()
 
 	candidates, err := selectCandidates(cluster, status)
 	if err != nil {
-		return []localityInfo{}, err
+		return []locality.Info{}, err
 	}
 
-	coordinators, err := chooseDistributedProcesses(cluster, candidates, coordinatorCount, processSelectionConstraint{
-		HardLimits: getHardLimits(cluster),
+	coordinators, err := locality.ChooseDistributedProcesses(cluster, candidates, coordinatorCount, locality.ProcessSelectionConstraint{
+		HardLimits: locality.GetHardLimits(cluster),
 	})
 
 	logger.Info("Current coordinators", "coordinators", coordinators)
@@ -171,7 +168,7 @@ func selectCoordinators(cluster *fdbv1beta2.FoundationDBCluster, status *fdbv1be
 		coordinatorStatus[getCoordinatorAddress(cluster, coordinator).String()] = false
 	}
 
-	hasValidCoordinators, allAddressesValid, err := checkCoordinatorValidity(cluster, status, coordinatorStatus)
+	hasValidCoordinators, allAddressesValid, err := locality.CheckCoordinatorValidity(logger, cluster, status, coordinatorStatus)
 	if err != nil {
 		return coordinators, err
 	}
@@ -187,7 +184,7 @@ func selectCoordinators(cluster *fdbv1beta2.FoundationDBCluster, status *fdbv1be
 	return coordinators, nil
 }
 
-func getCoordinatorAddress(cluster *fdbv1beta2.FoundationDBCluster, locality localityInfo) fdbv1beta2.ProcessAddress {
+func getCoordinatorAddress(cluster *fdbv1beta2.FoundationDBCluster, locality locality.Info) fdbv1beta2.ProcessAddress {
 	dnsName := locality.LocalityData[fdbv1beta2.FDBLocalityDNSNameKey]
 
 	address := locality.Address
diff --git a/controllers/choose_removals.go b/controllers/choose_removals.go
index 6d277e1e..29150c42 100644
--- a/controllers/choose_removals.go
+++ b/controllers/choose_removals.go
@@ -24,6 +24,8 @@ import (
 	"context"
 	"fmt"
 
+	"github.com/FoundationDB/fdb-kubernetes-operator/internal/locality"
+
 	corev1 "k8s.io/api/core/v1"
 
 	fdbv1beta2 "github.com/FoundationDB/fdb-kubernetes-operator/api/v1beta2"
@@ -60,10 +62,10 @@ func (c chooseRemovals) reconcile(ctx context.Context, r *FoundationDBClusterRec
 	if err != nil {
 		return &requeue{curError: err}
 	}
-	localityMap := make(map[string]localityInfo)
+	localityMap := make(map[string]locality.Info)
 	for _, process := range status.Cluster.Processes {
 		id := process.Locality[fdbv1beta2.FDBLocalityInstanceIDKey]
-		localityMap[id] = localityInfo{ID: id, Address: process.Address, LocalityData: process.Locality}
+		localityMap[id] = locality.Info{ID: id, Address: process.Address, LocalityData: process.Locality}
 	}
 
 	remainingProcessMap := make(map[string]bool, len(cluster.Status.ProcessGroups))
@@ -71,7 +73,7 @@ func (c chooseRemovals) reconcile(ctx context.Context, r *FoundationDBClusterRec
 	for _, processClass := range fdbv1beta2.ProcessClasses {
 		desiredCount := desiredCounts[processClass]
 		removedCount := currentCounts[processClass] - desiredCount
-		processClassLocality := make([]localityInfo, 0, currentCounts[processClass])
+		processClassLocality := make([]locality.Info, 0, currentCounts[processClass])
 
 		for _, processGroup := range cluster.Status.ProcessGroupsByProcessClass(processClass) {
 			if processGroup.IsMarkedForRemoval() {
@@ -87,7 +89,7 @@ func (c chooseRemovals) reconcile(ctx context.Context, r *FoundationDBClusterRec
 		if removedCount > 0 {
 			r.Recorder.Event(cluster, corev1.EventTypeNormal, "ShrinkingProcesses", fmt.Sprintf("Removing %d %s processes", removedCount, processClass))
 
-			remainingProcesses, err := chooseDistributedProcesses(cluster, processClassLocality, desiredCount, processSelectionConstraint{})
+			remainingProcesses, err := locality.ChooseDistributedProcesses(cluster, processClassLocality, desiredCount, locality.ProcessSelectionConstraint{})
 			if err != nil {
 				return &requeue{curError: err}
 			}
diff --git a/controllers/cluster_controller.go b/controllers/cluster_controller.go
index 5b6d8325..95d98902 100644
--- a/controllers/cluster_controller.go
+++ b/controllers/cluster_controller.go
@@ -23,11 +23,8 @@ package controllers
 import (
 	"context"
 	"encoding/json"
-	"errors"
 	"fmt"
-	"math"
 	"regexp"
-	"sort"
 	"time"
 
 	"github.com/FoundationDB/fdb-kubernetes-operator/pkg/fdbadminclient"
@@ -373,368 +370,6 @@ type clusterSubReconciler interface {
 	reconcile(ctx context.Context, r *FoundationDBClusterReconciler, cluster *fdbv1beta2.FoundationDBCluster) *requeue
 }
 
-// localityInfo captures information about a process for the purposes of
-// choosing diverse locality.
-type localityInfo struct {
-	// The process group ID
-	ID string
-
-	// The process's public address.
-	Address fdbv1beta2.ProcessAddress
-
-	// The locality map.
-	LocalityData map[string]string
-
-	Class fdbv1beta2.ProcessClass
-}
-
-// Sort processes by their priority and their ID.
-// We have to do this to ensure we get a deterministic result for selecting the candidates
-// otherwise we get a (nearly) random result since processes are stored in a map which is by definition
-// not sorted and doesn't return values in a stable way.
-func sortLocalities(cluster *fdbv1beta2.FoundationDBCluster, processes []localityInfo) {
-	// Sort the processes for ID to ensure we have a stable input
-	sort.SliceStable(processes, func(i, j int) bool {
-		p1 := cluster.GetClassCandidatePriority(processes[i].Class)
-		p2 := cluster.GetClassCandidatePriority(processes[j].Class)
-
-		// If both have the same priority sort them by the process ID
-		if p1 == p2 {
-			return processes[i].ID < processes[j].ID
-		}
-
-		// prefer processes with a higher priority
-		return p1 > p2
-	})
-}
-
-// localityInfoForProcess converts the process information from the JSON status
-// into locality info for selecting processes.
-func localityInfoForProcess(process fdbv1beta2.FoundationDBStatusProcessInfo, mainContainerTLS bool) (localityInfo, error) {
-	addresses, err := fdbv1beta2.ParseProcessAddressesFromCmdline(process.CommandLine)
-	if err != nil {
-		return localityInfo{}, err
-	}
-
-	var addr fdbv1beta2.ProcessAddress
-	// Iterate the addresses and set the expected address as process address
-	// e.g. if we want to use TLS set it to the tls address otherwise use the non-TLS.
-	for _, address := range addresses {
-		if address.Flags["tls"] == mainContainerTLS {
-			addr = address
-			break
-		}
-	}
-
-	return localityInfo{
-		ID:           process.Locality["instance_id"],
-		Address:      addr,
-		LocalityData: process.Locality,
-		Class:        process.ProcessClass,
-	}, nil
-}
-
-// localityInfoForProcess converts the process information from the sidecar's
-// context into locality info for selecting processes.
-func localityInfoFromSidecar(cluster *fdbv1beta2.FoundationDBCluster, client podclient.FdbPodClient) (localityInfo, error) {
-	substitutions, err := client.GetVariableSubstitutions()
-	if err != nil {
-		return localityInfo{}, err
-	}
-
-	if substitutions == nil {
-		return localityInfo{}, nil
-	}
-
-	// This locality information is only used during the initial cluster file generation.
-	// So it should be good to only use the first process address here.
-	// This has the implication that in the initial cluster file only the first processes will be used.
-	address := cluster.GetFullAddress(substitutions["FDB_PUBLIC_IP"], 1)
-	return localityInfo{
-		ID:      substitutions["FDB_INSTANCE_ID"],
-		Address: address,
-		LocalityData: map[string]string{
-			fdbv1beta2.FDBLocalityZoneIDKey:  substitutions["FDB_ZONE_ID"],
-			fdbv1beta2.FDBLocalityDNSNameKey: substitutions["FDB_DNS_NAME"],
-		},
-	}, nil
-}
-
-// notEnoughProcessesError is returned when we cannot recruit enough processes.
-type notEnoughProcessesError struct {
-	// desired defines the number of processes we wanted to recruit.
-	Desired int
-
-	// chosen defines the number of processes we were able to recruit.
-	Chosen int
-
-	// options defines the processes that we were selecting from.
-	Options []localityInfo
-}
-
-// Error formats an error message.
-func (err notEnoughProcessesError) Error() string {
-	return fmt.Sprintf("Could only select %d processes, but %d are required", err.Chosen, err.Desired)
-}
-
-// processSelectionConstraint defines constraints on how we choose processes
-// in chooseDistributedProcesses
-type processSelectionConstraint struct {
-	// Fields defines the locality fields we should consider when selecting
-	// processes.
-	Fields []string
-
-	// HardLimits defines a maximum number of processes to recruit on any single
-	// value for a given locality field.
-	HardLimits map[string]int
-}
-
-// chooseDistributedProcesses recruits a maximally well-distributed set
-// of processes from a set of potential candidates.
-func chooseDistributedProcesses(cluster *fdbv1beta2.FoundationDBCluster, processes []localityInfo, count int, constraint processSelectionConstraint) ([]localityInfo, error) {
-	chosen := make([]localityInfo, 0, count)
-	chosenIDs := make(map[string]bool, count)
-
-	fields := constraint.Fields
-	if len(fields) == 0 {
-		fields = []string{fdbv1beta2.FDBLocalityZoneIDKey, fdbv1beta2.FDBLocalityDCIDKey}
-	}
-
-	chosenCounts := make(map[string]map[string]int, len(fields))
-	hardLimits := make(map[string]int, len(fields))
-	currentLimits := make(map[string]int, len(fields))
-
-	if constraint.HardLimits != nil {
-		for field, limit := range constraint.HardLimits {
-			hardLimits[field] = limit
-		}
-	}
-
-	for _, field := range fields {
-		chosenCounts[field] = make(map[string]int)
-		if hardLimits[field] == 0 {
-			hardLimits[field] = count
-		}
-		currentLimits[field] = 1
-	}
-
-	// Sort the processes to ensure a deterministic result
-	sortLocalities(cluster, processes)
-
-	for len(chosen) < count {
-		choseAny := false
-
-		for _, process := range processes {
-			if chosenIDs[process.ID] {
-				continue
-			}
-
-			eligible := true
-			for _, field := range fields {
-				value := process.LocalityData[field]
-				if chosenCounts[field][value] >= currentLimits[field] {
-					eligible = false
-					break
-				}
-			}
-
-			if !eligible {
-				continue
-			}
-
-			chosen = append(chosen, process)
-			chosenIDs[process.ID] = true
-
-			choseAny = true
-
-			for _, field := range fields {
-				value := process.LocalityData[field]
-				chosenCounts[field][value]++
-			}
-
-			if len(chosen) == count {
-				break
-			}
-		}
-
-		if !choseAny {
-			incrementedLimits := false
-			for indexOfField := len(fields) - 1; indexOfField >= 0; indexOfField-- {
-				field := fields[indexOfField]
-				if currentLimits[field] < hardLimits[field] {
-					currentLimits[field]++
-					incrementedLimits = true
-					break
-				}
-			}
-			if !incrementedLimits {
-				return chosen, notEnoughProcessesError{Desired: count, Chosen: len(chosen), Options: processes}
-			}
-		}
-	}
-
-	return chosen, nil
-}
-
-func getHardLimits(cluster *fdbv1beta2.FoundationDBCluster) map[string]int {
-	if cluster.Spec.DatabaseConfiguration.UsableRegions <= 1 {
-		return map[string]int{fdbv1beta2.FDBLocalityZoneIDKey: 1}
-	}
-
-	// TODO (johscheuer): should we calculate that based on the number of DCs?
-	maxCoordinatorsPerDC := int(math.Floor(float64(cluster.DesiredCoordinatorCount()) / 2.0))
-
-	return map[string]int{fdbv1beta2.FDBLocalityZoneIDKey: 1, fdbv1beta2.FDBLocalityDCIDKey: maxCoordinatorsPerDC}
-}
-
-// checkCoordinatorValidity determines if the cluster's current coordinators
-// meet the fault tolerance requirements.
-//
-// The first return value will be whether the coordinators are valid.
-// The second return value will be whether the processes have their TLS flags
-// matching the cluster spec.
-// The third return value will hold any errors encountered when checking the
-// coordinators.
-func checkCoordinatorValidity(cluster *fdbv1beta2.FoundationDBCluster, status *fdbv1beta2.FoundationDBStatus, coordinatorStatus map[string]bool) (bool, bool, error) {
-	if len(coordinatorStatus) == 0 {
-		return false, false, errors.New("unable to get coordinator status")
-	}
-
-	curLog := log.WithValues("namespace", cluster.Namespace, "cluster", cluster.Name)
-
-	allAddressesValid := true
-	allEligible := true
-	allUsingCorrectAddress := true
-
-	coordinatorZones := make(map[string]int, len(coordinatorStatus))
-	coordinatorDCs := make(map[string]int, len(coordinatorStatus))
-	processGroups := make(map[string]*fdbv1beta2.ProcessGroupStatus)
-	for _, processGroup := range cluster.Status.ProcessGroups {
-		processGroups[processGroup.ProcessGroupID] = processGroup
-	}
-
-	for _, process := range status.Cluster.Processes {
-		pLogger := curLog.WithValues("process", process.Locality[fdbv1beta2.FDBLocalityInstanceIDKey])
-		if process.Address.IsEmpty() {
-			pLogger.Info("Skip process with empty address")
-			continue
-		}
-
-		if process.ProcessClass == fdbv1beta2.ProcessClassTest {
-			pLogger.Info("Ignoring test process")
-			continue
-		}
-
-		processGroupStatus := processGroups[process.Locality["instance_id"]]
-		pendingRemoval := processGroupStatus != nil && processGroupStatus.IsMarkedForRemoval()
-		if processGroupStatus != nil && cluster.SkipProcessGroup(processGroupStatus) {
-			log.Info("Skipping process group with pending Pod",
-				"namespace", cluster.Namespace,
-				"cluster", cluster.Name,
-				"processGroupID", process.Locality["instance_id"],
-				"class", process.ProcessClass)
-			continue
-		}
-
-		addresses, err := fdbv1beta2.ParseProcessAddressesFromCmdline(process.CommandLine)
-		if err != nil {
-			// We will end here in the error case when the address
-			// is not parsable e.g. no IP address is assigned.
-			allAddressesValid = false
-			// add: command_line
-			pLogger.Info("Could not parse address from command_line", "command_line", process.CommandLine)
-			continue
-		}
-
-		var ipAddress fdbv1beta2.ProcessAddress
-		for _, addr := range addresses {
-			if addr.Flags["tls"] == cluster.Spec.MainContainer.EnableTLS {
-				ipAddress = addr
-				break
-			}
-		}
-
-		coordinatorAddress := ""
-		_, isCoordinatorWithIP := coordinatorStatus[ipAddress.String()]
-		if isCoordinatorWithIP {
-			coordinatorAddress = ipAddress.String()
-		}
-
-		dnsName := process.Locality[fdbv1beta2.FDBLocalityDNSNameKey]
-		dnsAddress := fdbv1beta2.ProcessAddress{
-			StringAddress: dnsName,
-			Port:          ipAddress.Port,
-			Flags:         ipAddress.Flags,
-		}
-		_, isCoordinatorWithDNS := coordinatorStatus[dnsAddress.String()]
-
-		if !isCoordinatorWithDNS {
-			dnsAddress = ipAddress
-			dnsAddress.FromHostname = true
-			_, isCoordinatorWithDNS = coordinatorStatus[dnsAddress.String()]
-		}
-
-		if isCoordinatorWithDNS {
-			coordinatorAddress = dnsAddress.String()
-		}
-
-		if coordinatorAddress != "" && !process.Excluded && !pendingRemoval {
-			coordinatorStatus[coordinatorAddress] = true
-		}
-
-		if coordinatorAddress != "" {
-			coordinatorZones[process.Locality[fdbv1beta2.FDBLocalityZoneIDKey]]++
-			coordinatorDCs[process.Locality[fdbv1beta2.FDBLocalityDCIDKey]]++
-
-			if !cluster.IsEligibleAsCandidate(process.ProcessClass) {
-				pLogger.Info("Process class of process is not eligible as coordinator", "class", process.ProcessClass, "address", coordinatorAddress)
-				allEligible = false
-			}
-
-			useDNS := cluster.UseDNSInClusterFile() && dnsName != ""
-			if (isCoordinatorWithIP && useDNS) || (isCoordinatorWithDNS && !useDNS) {
-				pLogger.Info("Coordinator is not using the correct address type", "coordinatorList", coordinatorStatus, "address", coordinatorAddress, "expectingDNS", useDNS, "usingDNS", isCoordinatorWithDNS)
-				allUsingCorrectAddress = false
-			}
-		}
-
-		if ipAddress.IPAddress == nil {
-			pLogger.Info("Process has invalid IP address", "addresses", addresses)
-			allAddressesValid = false
-		}
-	}
-
-	desiredCount := cluster.DesiredCoordinatorCount()
-	hasEnoughZones := len(coordinatorZones) == desiredCount
-	if !hasEnoughZones {
-		curLog.Info("Cluster does not have coordinators in the correct number of zones", "desiredCount", desiredCount, "coordinatorZones", coordinatorZones)
-	}
-
-	var maxCoordinatorsPerDC int
-	hasEnoughDCs := true
-	if cluster.Spec.DatabaseConfiguration.UsableRegions > 1 {
-		maxCoordinatorsPerDC = int(math.Floor(float64(desiredCount) / 2.0))
-
-		for dc, count := range coordinatorDCs {
-			if count > maxCoordinatorsPerDC {
-				curLog.Info("Cluster has too many coordinators in a single DC", "DC", dc, "count", count, "max", maxCoordinatorsPerDC)
-				hasEnoughDCs = false
-			}
-		}
-	}
-
-	allHealthy := true
-	for address, healthy := range coordinatorStatus {
-		allHealthy = allHealthy && healthy
-
-		if !healthy {
-			curLog.Info("Cluster has an unhealthy coordinator", "address", address)
-		}
-	}
-
-	return hasEnoughDCs && hasEnoughZones && allHealthy && allUsingCorrectAddress && allEligible, allAddressesValid, nil
-}
-
 // newFdbPodClient builds a client for working with an FDB Pod
 func (r *FoundationDBClusterReconciler) newFdbPodClient(cluster *fdbv1beta2.FoundationDBCluster, pod *corev1.Pod) (podclient.FdbPodClient, error) {
 	return internal.NewFdbPodClient(cluster, pod, log.WithValues("namespace", cluster.Namespace, "cluster", cluster.Name, "pod", pod.Name), r.GetTimeout, r.PostTimeout)
diff --git a/controllers/generate_initial_cluster_file.go b/controllers/generate_initial_cluster_file.go
index b96e11c1..5b959c47 100644
--- a/controllers/generate_initial_cluster_file.go
+++ b/controllers/generate_initial_cluster_file.go
@@ -24,6 +24,8 @@ import (
 	"context"
 	"fmt"
 
+	"github.com/FoundationDB/fdb-kubernetes-operator/internal/locality"
+
 	"github.com/FoundationDB/fdb-kubernetes-operator/internal"
 
 	corev1 "k8s.io/api/core/v1"
@@ -100,31 +102,31 @@ func (g generateInitialClusterFile) reconcile(ctx context.Context, r *Foundation
 		}
 	}
 
-	processLocality := make([]localityInfo, 0, len(pods))
+	processLocality := make([]locality.Info, 0, len(pods))
 	for _, pod := range pods {
 		client, message := r.getPodClient(cluster, pod)
 		if client == nil {
 			return &requeue{message: message, delay: podSchedulingDelayDuration}
 		}
-		locality, err := localityInfoFromSidecar(cluster, client)
+		currentLocality, err := locality.InfoFromSidecar(cluster, client)
 		if err != nil {
 			return &requeue{curError: err}
 		}
-		if locality.ID == "" {
+		if currentLocality.ID == "" {
 			processGroupID := internal.GetProcessGroupIDFromMeta(cluster, pod.ObjectMeta)
 			logger.Info("Pod is ineligible to be a coordinator due to missing locality information", "processGroupID", processGroupID)
 			continue
 		}
-		processLocality = append(processLocality, locality)
+		processLocality = append(processLocality, currentLocality)
 	}
 
-	coordinators, err := chooseDistributedProcesses(cluster, processLocality, count, processSelectionConstraint{})
+	coordinators, err := locality.ChooseDistributedProcesses(cluster, processLocality, count, locality.ProcessSelectionConstraint{})
 	if err != nil {
 		return &requeue{curError: err}
 	}
 
-	for _, locality := range coordinators {
-		connectionString.Coordinators = append(connectionString.Coordinators, getCoordinatorAddress(cluster, locality).String())
+	for _, currentLocality := range coordinators {
+		connectionString.Coordinators = append(connectionString.Coordinators, getCoordinatorAddress(cluster, currentLocality).String())
 	}
 
 	cluster.Status.ConnectionString = connectionString.String()
@@ -133,5 +135,6 @@ func (g generateInitialClusterFile) reconcile(ctx context.Context, r *Foundation
 	if err != nil {
 		return &requeue{curError: err}
 	}
+
 	return nil
 }
diff --git a/controllers/update_status.go b/controllers/update_status.go
index 9d9d138d..3c1d1304 100644
--- a/controllers/update_status.go
+++ b/controllers/update_status.go
@@ -26,6 +26,8 @@ import (
 	"sort"
 	"time"
 
+	"github.com/FoundationDB/fdb-kubernetes-operator/internal/locality"
+
 	"github.com/FoundationDB/fdb-kubernetes-operator/pkg/podmanager"
 	"github.com/go-logr/logr"
 
@@ -81,6 +83,7 @@ func (updateStatus) reconcile(ctx context.Context, r *FoundationDBClusterReconci
 
 		databaseStatus, err = adminClient.GetStatus()
 		_ = adminClient.Close()
+
 		if err != nil {
 			if cluster.Status.Configured {
 				return &requeue{curError: err, delayedRequeue: true}
@@ -220,7 +223,7 @@ func (updateStatus) reconcile(ctx context.Context, r *FoundationDBClusterReconci
 			coordinatorStatus[coordinator.Address.String()] = false
 		}
 
-		coordinatorsValid, _, err := checkCoordinatorValidity(cluster, databaseStatus, coordinatorStatus)
+		coordinatorsValid, _, err := locality.CheckCoordinatorValidity(logger, cluster, databaseStatus, coordinatorStatus)
 		if err != nil {
 			return &requeue{curError: err, delayedRequeue: true}
 		}
@@ -321,16 +324,15 @@ func tryConnectionOptions(cluster *fdbv1beta2.FoundationDBCluster, r *Foundation
 		logger.Info("Attempting to get connection string from cluster", "connectionString", connectionString)
 		cluster.Status.ConnectionString = connectionString
 		adminClient, clientErr := r.getDatabaseClientProvider().GetAdminClient(cluster, r)
-
 		if clientErr != nil {
 			return originalConnectionString, clientErr
 		}
 
 		activeConnectionString, err := adminClient.GetConnectionString()
 
-		clientErr = adminClient.Close()
-		if clientErr != nil {
-			logger.V(1).Info("Could not close admin client")
+		closeErr := adminClient.Close()
+		if closeErr != nil {
+			logger.V(1).Info("Could not close admin client", "error", closeErr)
 		}
 
 		if err == nil {
diff --git a/internal/locality/locality.go b/internal/locality/locality.go
new file mode 100644
index 00000000..ba56f0db
--- /dev/null
+++ b/internal/locality/locality.go
@@ -0,0 +1,391 @@
+/*
+ * locality.go
+ *
+ * This source file is part of the FoundationDB open source project
+ *
+ * Copyright 2022 Apple Inc. and the FoundationDB project authors
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package locality
+
+import (
+	"errors"
+	"fmt"
+	"math"
+	"sort"
+
+	fdbv1beta2 "github.com/FoundationDB/fdb-kubernetes-operator/api/v1beta2"
+	"github.com/FoundationDB/fdb-kubernetes-operator/pkg/podclient"
+	"github.com/go-logr/logr"
+)
+
+// Info captures information about a process for the purposes of
+// choosing diverse locality.
+type Info struct {
+	// The process group ID
+	ID string
+
+	// The process's public address.
+	Address fdbv1beta2.ProcessAddress
+
+	// The locality map.
+	LocalityData map[string]string
+
+	Class fdbv1beta2.ProcessClass
+}
+
+// Sort processes by their priority and their ID.
+// We have to do this to ensure we get a deterministic result for selecting the candidates
+// otherwise we get a (nearly) random result since processes are stored in a map which is by definition
+// not sorted and doesn't return values in a stable way.
+func sortLocalities(cluster *fdbv1beta2.FoundationDBCluster, processes []Info) {
+	// Sort the processes for ID to ensure we have a stable input
+	sort.SliceStable(processes, func(i, j int) bool {
+		p1 := cluster.GetClassCandidatePriority(processes[i].Class)
+		p2 := cluster.GetClassCandidatePriority(processes[j].Class)
+
+		// If both have the same priority sort them by the process ID
+		if p1 == p2 {
+			return processes[i].ID < processes[j].ID
+		}
+
+		// prefer processes with a higher priority
+		return p1 > p2
+	})
+}
+
+// InfoForProcess converts the process information from the JSON status
+// into locality info for selecting processes.
+func InfoForProcess(process fdbv1beta2.FoundationDBStatusProcessInfo, mainContainerTLS bool) (Info, error) {
+	addresses, err := fdbv1beta2.ParseProcessAddressesFromCmdline(process.CommandLine)
+	if err != nil {
+		return Info{}, err
+	}
+
+	var addr fdbv1beta2.ProcessAddress
+	// Iterate the addresses and set the expected address as process address
+	// e.g. if we want to use TLS set it to the tls address otherwise use the non-TLS.
+	for _, address := range addresses {
+		if address.Flags["tls"] == mainContainerTLS {
+			addr = address
+			break
+		}
+	}
+
+	return Info{
+		ID:           process.Locality[fdbv1beta2.FDBLocalityInstanceIDKey],
+		Address:      addr,
+		LocalityData: process.Locality,
+		Class:        process.ProcessClass,
+	}, nil
+}
+
+// InfoFromSidecar converts the process information from the sidecar's
+// context into locality info for selecting processes.
+func InfoFromSidecar(cluster *fdbv1beta2.FoundationDBCluster, client podclient.FdbPodClient) (Info, error) {
+	substitutions, err := client.GetVariableSubstitutions()
+	if err != nil {
+		return Info{}, err
+	}
+
+	if substitutions == nil {
+		return Info{}, nil
+	}
+
+	// This locality information is only used during the initial cluster file generation.
+	// So it should be good to only use the first process address here.
+	// This has the implication that in the initial cluster file only the first processes will be used.
+	return Info{
+		ID:      substitutions["FDB_INSTANCE_ID"],
+		Address: cluster.GetFullAddress(substitutions["FDB_PUBLIC_IP"], 1),
+		LocalityData: map[string]string{
+			fdbv1beta2.FDBLocalityZoneIDKey:  substitutions["FDB_ZONE_ID"],
+			fdbv1beta2.FDBLocalityDNSNameKey: substitutions["FDB_DNS_NAME"],
+		},
+	}, nil
+}
+
+// notEnoughProcessesError is returned when we cannot recruit enough processes.
+type notEnoughProcessesError struct {
+	// desired defines the number of processes we wanted to recruit.
+	Desired int
+
+	// chosen defines the number of processes we were able to recruit.
+	Chosen int
+
+	// options defines the processes that we were selecting from.
+	Options []Info
+}
+
+// Error formats an error message.
+func (err notEnoughProcessesError) Error() string {
+	return fmt.Sprintf("Could only select %d processes, but %d are required", err.Chosen, err.Desired)
+}
+
+// ProcessSelectionConstraint defines constraints on how we choose processes
+// in chooseDistributedProcesses
+type ProcessSelectionConstraint struct {
+	// Fields defines the locality fields we should consider when selecting
+	// processes.
+	Fields []string
+
+	// HardLimits defines a maximum number of processes to recruit on any single
+	// value for a given locality field.
+	HardLimits map[string]int
+}
+
+// ChooseDistributedProcesses recruits a maximally well-distributed set
+// of processes from a set of potential candidates.
+func ChooseDistributedProcesses(cluster *fdbv1beta2.FoundationDBCluster, processes []Info, count int, constraint ProcessSelectionConstraint) ([]Info, error) {
+	chosen := make([]Info, 0, count)
+	chosenIDs := make(map[string]bool, count)
+
+	fields := constraint.Fields
+	if len(fields) == 0 {
+		fields = []string{fdbv1beta2.FDBLocalityZoneIDKey, fdbv1beta2.FDBLocalityDCIDKey}
+	}
+
+	chosenCounts := make(map[string]map[string]int, len(fields))
+	hardLimits := make(map[string]int, len(fields))
+	currentLimits := make(map[string]int, len(fields))
+
+	if constraint.HardLimits != nil {
+		for field, limit := range constraint.HardLimits {
+			hardLimits[field] = limit
+		}
+	}
+
+	for _, field := range fields {
+		chosenCounts[field] = make(map[string]int)
+		if hardLimits[field] == 0 {
+			hardLimits[field] = count
+		}
+		currentLimits[field] = 1
+	}
+
+	// Sort the processes to ensure a deterministic result
+	sortLocalities(cluster, processes)
+
+	for len(chosen) < count {
+		choseAny := false
+
+		for _, process := range processes {
+			if chosenIDs[process.ID] {
+				continue
+			}
+
+			eligible := true
+			for _, field := range fields {
+				value := process.LocalityData[field]
+				if chosenCounts[field][value] >= currentLimits[field] {
+					eligible = false
+					break
+				}
+			}
+
+			if !eligible {
+				continue
+			}
+
+			chosen = append(chosen, process)
+			chosenIDs[process.ID] = true
+
+			choseAny = true
+
+			for _, field := range fields {
+				value := process.LocalityData[field]
+				chosenCounts[field][value]++
+			}
+
+			if len(chosen) == count {
+				break
+			}
+		}
+
+		if !choseAny {
+			incrementedLimits := false
+			for indexOfField := len(fields) - 1; indexOfField >= 0; indexOfField-- {
+				field := fields[indexOfField]
+				if currentLimits[field] < hardLimits[field] {
+					currentLimits[field]++
+					incrementedLimits = true
+					break
+				}
+			}
+			if !incrementedLimits {
+				return chosen, notEnoughProcessesError{Desired: count, Chosen: len(chosen), Options: processes}
+			}
+		}
+	}
+
+	return chosen, nil
+}
+
+// GetHardLimits returns the distribution of localities.
+func GetHardLimits(cluster *fdbv1beta2.FoundationDBCluster) map[string]int {
+	if cluster.Spec.DatabaseConfiguration.UsableRegions <= 1 {
+		return map[string]int{fdbv1beta2.FDBLocalityZoneIDKey: 1}
+	}
+
+	// TODO (johscheuer): should we calculate that based on the number of DCs?
+	maxCoordinatorsPerDC := int(math.Floor(float64(cluster.DesiredCoordinatorCount()) / 2.0))
+
+	return map[string]int{fdbv1beta2.FDBLocalityZoneIDKey: 1, fdbv1beta2.FDBLocalityDCIDKey: maxCoordinatorsPerDC}
+}
+
+// CheckCoordinatorValidity determines if the cluster's current coordinators
+// meet the fault tolerance requirements.
+//
+// The first return value will be whether the coordinators are valid.
+// The second return value will be whether the processes have their TLS flags
+// matching the cluster spec.
+// The third return value will hold any errors encountered when checking the
+// coordinators.
+func CheckCoordinatorValidity(logger logr.Logger, cluster *fdbv1beta2.FoundationDBCluster, status *fdbv1beta2.FoundationDBStatus, coordinatorStatus map[string]bool) (bool, bool, error) {
+	if len(coordinatorStatus) == 0 {
+		return false, false, errors.New("unable to get coordinator status")
+	}
+
+	allAddressesValid := true
+	allEligible := true
+	allUsingCorrectAddress := true
+
+	coordinatorZones := make(map[string]int, len(coordinatorStatus))
+	coordinatorDCs := make(map[string]int, len(coordinatorStatus))
+	processGroups := make(map[string]*fdbv1beta2.ProcessGroupStatus)
+	for _, processGroup := range cluster.Status.ProcessGroups {
+		processGroups[processGroup.ProcessGroupID] = processGroup
+	}
+
+	for _, process := range status.Cluster.Processes {
+		pLogger := logger.WithValues("process", process.Locality[fdbv1beta2.FDBLocalityInstanceIDKey])
+		if process.Address.IsEmpty() {
+			pLogger.Info("Skip process with empty address")
+			continue
+		}
+
+		if process.ProcessClass == fdbv1beta2.ProcessClassTest {
+			pLogger.Info("Ignoring tester process")
+			continue
+		}
+
+		processGroupStatus := processGroups[process.Locality[fdbv1beta2.FDBLocalityInstanceIDKey]]
+		pendingRemoval := processGroupStatus != nil && processGroupStatus.IsMarkedForRemoval()
+		if processGroupStatus != nil && cluster.SkipProcessGroup(processGroupStatus) {
+			pLogger.Info("Skipping process group with pending Pod",
+				"namespace", cluster.Namespace,
+				"cluster", cluster.Name,
+				"processGroupID", process.Locality[fdbv1beta2.FDBLocalityInstanceIDKey],
+				"class", process.ProcessClass)
+			continue
+		}
+
+		addresses, err := fdbv1beta2.ParseProcessAddressesFromCmdline(process.CommandLine)
+		if err != nil {
+			// We will end here in the error case when the address
+			// is not parsable e.g. no IP address is assigned.
+			allAddressesValid = false
+			// add: command_line
+			pLogger.Info("Could not parse address from command_line", "command_line", process.CommandLine)
+			continue
+		}
+
+		var ipAddress fdbv1beta2.ProcessAddress
+		for _, addr := range addresses {
+			if addr.Flags["tls"] == cluster.Spec.MainContainer.EnableTLS {
+				ipAddress = addr
+				break
+			}
+		}
+
+		coordinatorAddress := ""
+		_, isCoordinatorWithIP := coordinatorStatus[ipAddress.String()]
+		if isCoordinatorWithIP {
+			coordinatorAddress = ipAddress.String()
+		}
+
+		dnsName := process.Locality[fdbv1beta2.FDBLocalityDNSNameKey]
+		dnsAddress := fdbv1beta2.ProcessAddress{
+			StringAddress: dnsName,
+			Port:          ipAddress.Port,
+			Flags:         ipAddress.Flags,
+		}
+		_, isCoordinatorWithDNS := coordinatorStatus[dnsAddress.String()]
+
+		if !isCoordinatorWithDNS {
+			dnsAddress = ipAddress
+			dnsAddress.FromHostname = true
+			_, isCoordinatorWithDNS = coordinatorStatus[dnsAddress.String()]
+		}
+
+		if isCoordinatorWithDNS {
+			coordinatorAddress = dnsAddress.String()
+		}
+
+		if coordinatorAddress != "" && !process.Excluded && !pendingRemoval {
+			coordinatorStatus[coordinatorAddress] = true
+		}
+
+		if coordinatorAddress != "" {
+			coordinatorZones[process.Locality[fdbv1beta2.FDBLocalityZoneIDKey]]++
+			coordinatorDCs[process.Locality[fdbv1beta2.FDBLocalityDCIDKey]]++
+
+			if !cluster.IsEligibleAsCandidate(process.ProcessClass) {
+				pLogger.Info("Process class of process is not eligible as coordinator", "class", process.ProcessClass, "address", coordinatorAddress)
+				allEligible = false
+			}
+
+			useDNS := cluster.UseDNSInClusterFile() && dnsName != ""
+			if (isCoordinatorWithIP && useDNS) || (isCoordinatorWithDNS && !useDNS) {
+				pLogger.Info("Coordinator is not using the correct address type", "coordinatorList", coordinatorStatus, "address", coordinatorAddress, "expectingDNS", useDNS, "usingDNS", isCoordinatorWithDNS)
+				allUsingCorrectAddress = false
+			}
+		}
+
+		if ipAddress.IPAddress == nil {
+			pLogger.Info("Process has invalid IP address", "addresses", addresses)
+			allAddressesValid = false
+		}
+	}
+
+	desiredCount := cluster.DesiredCoordinatorCount()
+	hasEnoughZones := len(coordinatorZones) == desiredCount
+	if !hasEnoughZones {
+		logger.Info("Cluster does not have coordinators in the correct number of zones", "desiredCount", desiredCount, "coordinatorZones", coordinatorZones)
+	}
+
+	var maxCoordinatorsPerDC int
+	hasEnoughDCs := true
+	if cluster.Spec.DatabaseConfiguration.UsableRegions > 1 {
+		maxCoordinatorsPerDC = int(math.Floor(float64(desiredCount) / 2.0))
+
+		for dc, count := range coordinatorDCs {
+			if count > maxCoordinatorsPerDC {
+				logger.Info("Cluster has too many coordinators in a single DC", "DC", dc, "count", count, "max", maxCoordinatorsPerDC)
+				hasEnoughDCs = false
+			}
+		}
+	}
+
+	allHealthy := true
+	for address, healthy := range coordinatorStatus {
+		if !healthy {
+			allHealthy = false
+			logger.Info("Cluster has an unhealthy coordinator", "address", address)
+		}
+	}
+
+	return hasEnoughDCs && hasEnoughZones && allHealthy && allUsingCorrectAddress && allEligible, allAddressesValid, nil
+}


Test patch:
diff --git a/api/v1beta2/foundationdb_status_test.go b/api/v1beta2/foundationdb_status_test.go
index 843a087f..8e566319 100644
--- a/api/v1beta2/foundationdb_status_test.go
+++ b/api/v1beta2/foundationdb_status_test.go
@@ -71,6 +71,7 @@ var _ = Describe("FoundationDBStatus", func() {
 				},
 				Cluster: FoundationDBStatusClusterInfo{
 					IncompatibleConnections: []string{},
+					ConnectionString:        "sample_cluster:JLjCjL6Vp3kWoIfHJeDZMhYqPBb1bIZr@10.1.38.94:4501,10.1.38.102:4501,10.1.38.104:4501",
 					FaultTolerance: FaultTolerance{
 						MaxZoneFailuresWithoutLosingAvailability: 1,
 						MaxZoneFailuresWithoutLosingData:         1,
@@ -464,6 +465,7 @@ var _ = Describe("FoundationDBStatus", func() {
 	When("parsing the status json with a 7.1.0-rc1 cluster", func() {
 		status := FoundationDBStatusClusterInfo{
 			IncompatibleConnections: []string{},
+			ConnectionString:        "test_cluster:aHeD9ocNXOUxi0dyzU3k7Bhg53SpyrBV@10.1.18.253:4501,10.1.18.254:4501,10.1.19.0:4501",
 			DatabaseConfiguration: DatabaseConfiguration{
 				RedundancyMode:  "double",
 				StorageEngine:   StorageEngineSSD2,
diff --git a/controllers/change_coordinators_test.go b/controllers/change_coordinators_test.go
index 2fbf6faa..2d279ccf 100644
--- a/controllers/change_coordinators_test.go
+++ b/controllers/change_coordinators_test.go
@@ -27,6 +27,9 @@ import (
 	"net"
 	"strings"
 
+	"github.com/FoundationDB/fdb-kubernetes-operator/internal/locality"
+	"github.com/go-logr/logr"
+
 	"github.com/FoundationDB/fdb-kubernetes-operator/internal"
 	corev1 "k8s.io/api/core/v1"
 	"k8s.io/utils/pointer"
@@ -64,14 +67,14 @@ var _ = Describe("Change coordinators", func() {
 	Describe("selectCoordinators", func() {
 		Context("with a single FDB cluster", func() {
 			var status *fdbv1beta2.FoundationDBStatus
-			var candidates []localityInfo
+			var candidates []locality.Info
 
 			JustBeforeEach(func() {
 				var err error
 				status, err = adminClient.GetStatus()
 				Expect(err).NotTo(HaveOccurred())
 
-				candidates, err = selectCoordinators(cluster, status)
+				candidates, err = selectCoordinators(logr.Discard(), cluster, status)
 				Expect(err).NotTo(HaveOccurred())
 			})
 
@@ -171,7 +174,7 @@ var _ = Describe("Change coordinators", func() {
 					initialCandidates := candidates
 
 					for i := 0; i < 100; i++ {
-						newCandidates, err := selectCoordinators(cluster, status)
+						newCandidates, err := selectCoordinators(logr.Discard(), cluster, status)
 						Expect(err).NotTo(HaveOccurred())
 						Expect(newCandidates).To(Equal(initialCandidates))
 					}
@@ -202,7 +205,7 @@ var _ = Describe("Change coordinators", func() {
 
 		When("Using a HA clusters", func() {
 			var status *fdbv1beta2.FoundationDBStatus
-			var candidates []localityInfo
+			var candidates []locality.Info
 			var excludes []string
 			var removals []string
 			var dcCnt int
@@ -211,7 +214,7 @@ var _ = Describe("Change coordinators", func() {
 
 			BeforeEach(func() {
 				// ensure a clean state
-				candidates = []localityInfo{}
+				candidates = []locality.Info{}
 				excludes = []string{}
 				removals = []string{}
 				shouldFail = false
@@ -229,7 +232,7 @@ var _ = Describe("Change coordinators", func() {
 				// generate status for 2 dcs and 1 sate
 				status.Cluster.Processes = generateProcessInfo(dcCnt, satCnt, excludes)
 
-				candidates, err = selectCoordinators(cluster, status)
+				candidates, err = selectCoordinators(logr.Discard(), cluster, status)
 				if shouldFail {
 					Expect(err).To(HaveOccurred())
 				} else {
@@ -401,7 +404,7 @@ var _ = Describe("Change coordinators", func() {
 						initialCandidates := candidates
 
 						for i := 0; i < 100; i++ {
-							newCandidates, err := selectCoordinators(cluster, status)
+							newCandidates, err := selectCoordinators(logr.Discard(), cluster, status)
 							Expect(err).NotTo(HaveOccurred())
 							Expect(newCandidates).To(Equal(initialCandidates))
 						}
@@ -599,7 +602,7 @@ var _ = Describe("Change coordinators", func() {
 						initialCandidates := candidates
 
 						for i := 0; i < 100; i++ {
-							newCandidates, err := selectCoordinators(cluster, status)
+							newCandidates, err := selectCoordinators(logr.Discard(), cluster, status)
 							Expect(err).NotTo(HaveOccurred())
 							Expect(newCandidates).To(Equal(initialCandidates))
 						}
@@ -607,104 +610,6 @@ var _ = Describe("Change coordinators", func() {
 				})
 			})
 		})
-
-		// TODO add test case for multi KC
-
-		When("Sorting the localities", func() {
-			var localities []localityInfo
-
-			BeforeEach(func() {
-				localities = []localityInfo{
-					{
-						ID:    "storage-1",
-						Class: fdbv1beta2.ProcessClassStorage,
-					},
-					{
-						ID:    "tlog-1",
-						Class: fdbv1beta2.ProcessClassTransaction,
-					},
-					{
-						ID:    "log-1",
-						Class: fdbv1beta2.ProcessClassLog,
-					},
-					{
-						ID:    "storage-51",
-						Class: fdbv1beta2.ProcessClassStorage,
-					},
-				}
-			})
-
-			When("no other preferences are defined", func() {
-				BeforeEach(func() {
-					cluster.Spec.CoordinatorSelection = []fdbv1beta2.CoordinatorSelectionSetting{}
-				})
-
-				It("should sort the localities based on the IDs", func() {
-					sortLocalities(cluster, localities)
-
-					Expect(localities[0].Class).To(Equal(fdbv1beta2.ProcessClassLog))
-					Expect(localities[0].ID).To(Equal("log-1"))
-					Expect(localities[1].Class).To(Equal(fdbv1beta2.ProcessClassStorage))
-					Expect(localities[1].ID).To(Equal("storage-1"))
-					Expect(localities[2].Class).To(Equal(fdbv1beta2.ProcessClassStorage))
-					Expect(localities[2].ID).To(Equal("storage-51"))
-					Expect(localities[3].Class).To(Equal(fdbv1beta2.ProcessClassTransaction))
-					Expect(localities[3].ID).To(Equal("tlog-1"))
-				})
-			})
-
-			When("when the storage class is preferred", func() {
-				BeforeEach(func() {
-					cluster.Spec.CoordinatorSelection = []fdbv1beta2.CoordinatorSelectionSetting{
-						{
-							ProcessClass: fdbv1beta2.ProcessClassStorage,
-							Priority:     0,
-						},
-					}
-				})
-
-				It("should sort the localities based on the provided config", func() {
-					sortLocalities(cluster, localities)
-
-					Expect(localities[0].Class).To(Equal(fdbv1beta2.ProcessClassStorage))
-					Expect(localities[0].ID).To(Equal("storage-1"))
-					Expect(localities[1].Class).To(Equal(fdbv1beta2.ProcessClassStorage))
-					Expect(localities[1].ID).To(Equal("storage-51"))
-					Expect(localities[2].Class).To(Equal(fdbv1beta2.ProcessClassLog))
-					Expect(localities[2].ID).To(Equal("log-1"))
-					Expect(localities[3].Class).To(Equal(fdbv1beta2.ProcessClassTransaction))
-					Expect(localities[3].ID).To(Equal("tlog-1"))
-				})
-			})
-
-			When("when the storage class is preferred over transaction class", func() {
-				BeforeEach(func() {
-					cluster.Spec.CoordinatorSelection = []fdbv1beta2.CoordinatorSelectionSetting{
-						{
-							ProcessClass: fdbv1beta2.ProcessClassStorage,
-							Priority:     1,
-						},
-						{
-							ProcessClass: fdbv1beta2.ProcessClassTransaction,
-							Priority:     0,
-						},
-					}
-				})
-
-				It("should sort the localities based on the provided config", func() {
-					sortLocalities(cluster, localities)
-
-					Expect(localities[0].Class).To(Equal(fdbv1beta2.ProcessClassStorage))
-					Expect(localities[0].ID).To(Equal("storage-1"))
-					Expect(localities[1].Class).To(Equal(fdbv1beta2.ProcessClassStorage))
-					Expect(localities[1].ID).To(Equal("storage-51"))
-					Expect(localities[2].Class).To(Equal(fdbv1beta2.ProcessClassTransaction))
-					Expect(localities[2].ID).To(Equal("tlog-1"))
-					Expect(localities[3].Class).To(Equal(fdbv1beta2.ProcessClassLog))
-					Expect(localities[3].ID).To(Equal("log-1"))
-				})
-			})
-		})
 	})
 
 	Describe("reconcile", func() {
diff --git a/controllers/cluster_controller_test.go b/controllers/cluster_controller_test.go
index 393df336..ab413029 100644
--- a/controllers/cluster_controller_test.go
+++ b/controllers/cluster_controller_test.go
@@ -24,13 +24,11 @@ import (
 	"bytes"
 	"context"
 	"fmt"
-	"net"
 	"regexp"
 	"sort"
 	"strings"
 	"time"
 
-	"github.com/FoundationDB/fdb-kubernetes-operator/pkg/fdbadminclient"
 	"github.com/FoundationDB/fdb-kubernetes-operator/pkg/podmanager"
 
 	"k8s.io/utils/pointer"
@@ -3311,532 +3309,6 @@ var _ = Describe("cluster_controller", func() {
 		})
 	})
 
-	Describe("chooseDistributedProcesses", func() {
-		var candidates []localityInfo
-		var result []localityInfo
-		var err error
-
-		Context("with a flat set of processes", func() {
-			BeforeEach(func() {
-				candidates = []localityInfo{
-					{ID: "p1", LocalityData: map[string]string{"zoneid": "z1"}},
-					{ID: "p2", LocalityData: map[string]string{"zoneid": "z1"}},
-					{ID: "p3", LocalityData: map[string]string{"zoneid": "z2"}},
-					{ID: "p4", LocalityData: map[string]string{"zoneid": "z3"}},
-					{ID: "p5", LocalityData: map[string]string{"zoneid": "z2"}},
-					{ID: "p6", LocalityData: map[string]string{"zoneid": "z4"}},
-					{ID: "p7", LocalityData: map[string]string{"zoneid": "z5"}},
-				}
-				result, err = chooseDistributedProcesses(cluster, candidates, 5, processSelectionConstraint{})
-				Expect(err).NotTo(HaveOccurred())
-			})
-
-			It("should recruit the processes across multiple zones", func() {
-				Expect(len(result)).To(Equal(5))
-				Expect(result[0].ID).To(Equal("p1"))
-				Expect(result[1].ID).To(Equal("p3"))
-				Expect(result[2].ID).To(Equal("p4"))
-				Expect(result[3].ID).To(Equal("p6"))
-				Expect(result[4].ID).To(Equal("p7"))
-			})
-		})
-
-		Context("with fewer zones than desired processes", func() {
-			BeforeEach(func() {
-				candidates = []localityInfo{
-					{ID: "p1", LocalityData: map[string]string{"zoneid": "z1"}},
-					{ID: "p2", LocalityData: map[string]string{"zoneid": "z1"}},
-					{ID: "p3", LocalityData: map[string]string{"zoneid": "z2"}},
-					{ID: "p4", LocalityData: map[string]string{"zoneid": "z3"}},
-					{ID: "p5", LocalityData: map[string]string{"zoneid": "z2"}},
-					{ID: "p6", LocalityData: map[string]string{"zoneid": "z4"}},
-				}
-			})
-
-			Context("with no hard limit", func() {
-				It("should only re-use zones as necessary", func() {
-					result, err = chooseDistributedProcesses(cluster, candidates, 5, processSelectionConstraint{})
-					Expect(err).NotTo(HaveOccurred())
-
-					Expect(len(result)).To(Equal(5))
-					Expect(result[0].ID).To(Equal("p1"))
-					Expect(result[1].ID).To(Equal("p3"))
-					Expect(result[2].ID).To(Equal("p4"))
-					Expect(result[3].ID).To(Equal("p6"))
-					Expect(result[4].ID).To(Equal("p2"))
-				})
-			})
-
-			Context("with a hard limit", func() {
-				It("should give an error", func() {
-					result, err = chooseDistributedProcesses(cluster, candidates, 5, processSelectionConstraint{
-						HardLimits: map[string]int{"zoneid": 1},
-					})
-					Expect(err).To(HaveOccurred())
-					Expect(err.Error()).To(Equal("Could only select 4 processes, but 5 are required"))
-				})
-			})
-		})
-
-		Context("with multiple data centers", func() {
-			BeforeEach(func() {
-				candidates = []localityInfo{
-					{ID: "p1", LocalityData: map[string]string{"zoneid": "z1", "dcid": "dc1"}},
-					{ID: "p2", LocalityData: map[string]string{"zoneid": "z1", "dcid": "dc1"}},
-					{ID: "p3", LocalityData: map[string]string{"zoneid": "z2", "dcid": "dc1"}},
-					{ID: "p4", LocalityData: map[string]string{"zoneid": "z3", "dcid": "dc1"}},
-					{ID: "p5", LocalityData: map[string]string{"zoneid": "z2", "dcid": "dc1"}},
-					{ID: "p6", LocalityData: map[string]string{"zoneid": "z4", "dcid": "dc1"}},
-					{ID: "p7", LocalityData: map[string]string{"zoneid": "z5", "dcid": "dc1"}},
-					{ID: "p8", LocalityData: map[string]string{"zoneid": "z6", "dcid": "dc2"}},
-					{ID: "p9", LocalityData: map[string]string{"zoneid": "z7", "dcid": "dc2"}},
-					{ID: "p10", LocalityData: map[string]string{"zoneid": "z8", "dcid": "dc2"}},
-				}
-			})
-
-			Context("with the default constraints", func() {
-				BeforeEach(func() {
-					result, err = chooseDistributedProcesses(cluster, candidates, 5, processSelectionConstraint{})
-					Expect(err).NotTo(HaveOccurred())
-				})
-
-				It("should recruit the processes across multiple zones and data centers", func() {
-					Expect(len(result)).To(Equal(5))
-					Expect(result[0].ID).To(Equal("p1"))
-					Expect(result[1].ID).To(Equal("p10"))
-					Expect(result[2].ID).To(Equal("p3"))
-					Expect(result[3].ID).To(Equal("p8"))
-					Expect(result[4].ID).To(Equal("p4"))
-				})
-			})
-
-			Context("when only distributing across data centers", func() {
-				BeforeEach(func() {
-					result, err = chooseDistributedProcesses(cluster, candidates, 5, processSelectionConstraint{
-						Fields: []string{"dcid"},
-					})
-					Expect(err).NotTo(HaveOccurred())
-				})
-
-				It("should recruit the processes across data centers", func() {
-					Expect(len(result)).To(Equal(5))
-					Expect(result[0].ID).To(Equal("p1"))
-					Expect(result[1].ID).To(Equal("p10"))
-					Expect(result[2].ID).To(Equal("p2"))
-					Expect(result[3].ID).To(Equal("p8"))
-					Expect(result[4].ID).To(Equal("p3"))
-				})
-			})
-		})
-	})
-
-	Describe("checkCoordinatorValidity", func() {
-		var status *fdbv1beta2.FoundationDBStatus
-		var adminClient fdbadminclient.AdminClient
-		var err error
-
-		BeforeEach(func() {
-			err = k8sClient.Create(context.TODO(), cluster)
-			Expect(err).NotTo(HaveOccurred())
-
-			result, err := reconcileCluster(cluster)
-			Expect(err).NotTo(HaveOccurred())
-			Expect(result.Requeue).To(BeFalse())
-
-			generations, err := reloadClusterGenerations(cluster)
-			Expect(err).NotTo(HaveOccurred())
-			Expect(generations.Reconciled).To(Equal(int64(1)))
-
-			adminClient, err = newMockAdminClient(cluster, k8sClient)
-			Expect(err).NotTo(HaveOccurred())
-
-			status, err = adminClient.GetStatus()
-			Expect(err).NotTo(HaveOccurred())
-		})
-
-		Context("with the default configuration", func() {
-			It("should report the coordinators as valid", func() {
-				coordinatorStatus := make(map[string]bool, len(status.Client.Coordinators.Coordinators))
-				for _, coordinator := range status.Client.Coordinators.Coordinators {
-					coordinatorStatus[coordinator.Address.String()] = false
-				}
-
-				coordinatorsValid, addressesValid, err := checkCoordinatorValidity(cluster, status, coordinatorStatus)
-				Expect(coordinatorsValid).To(BeTrue())
-				Expect(addressesValid).To(BeTrue())
-				Expect(err).To(BeNil())
-			})
-		})
-
-		Context("with a tester process without a command line field", func() {
-			It("should ignore the test process and report the coordinators as valid", func() {
-				testProcess := fdbv1beta2.FoundationDBStatusProcessInfo{
-					Address: fdbv1beta2.ProcessAddress{
-						IPAddress: net.ParseIP("9.9.9.9"),
-					},
-					ProcessClass: fdbv1beta2.ProcessClassTest,
-				}
-				Expect(testProcess.Address.IsEmpty()).To(BeFalse())
-				status.Cluster.Processes[string(fdbv1beta2.ProcessClassTest)] = testProcess
-
-				coordinatorStatus := make(map[string]bool, len(status.Client.Coordinators.Coordinators))
-				for _, coordinator := range status.Client.Coordinators.Coordinators {
-					coordinatorStatus[coordinator.Address.String()] = false
-				}
-
-				coordinatorsValid, addressesValid, err := checkCoordinatorValidity(cluster, status, coordinatorStatus)
-				Expect(coordinatorsValid).To(BeTrue())
-				Expect(addressesValid).To(BeTrue())
-				Expect(err).To(BeNil())
-			})
-		})
-
-		Context("with too few coordinators", func() {
-			BeforeEach(func() {
-				status.Client.Coordinators.Coordinators = status.Client.Coordinators.Coordinators[0:2]
-			})
-
-			It("should report the coordinators as not valid", func() {
-				coordinatorStatus := make(map[string]bool, len(status.Client.Coordinators.Coordinators))
-				for _, coordinator := range status.Client.Coordinators.Coordinators {
-					coordinatorStatus[coordinator.Address.String()] = false
-				}
-
-				coordinatorsValid, addressesValid, err := checkCoordinatorValidity(cluster, status, coordinatorStatus)
-				Expect(coordinatorsValid).To(BeFalse())
-				Expect(addressesValid).To(BeTrue())
-				Expect(err).To(BeNil())
-			})
-		})
-
-		Context("with too few zones", func() {
-			BeforeEach(func() {
-				zone := ""
-				for _, process := range status.Cluster.Processes {
-					if process.Address.Equal(status.Client.Coordinators.Coordinators[0].Address) {
-						zone = process.Locality[fdbv1beta2.FDBLocalityZoneIDKey]
-					}
-				}
-				for _, process := range status.Cluster.Processes {
-					if process.Address.Equal(status.Client.Coordinators.Coordinators[1].Address) {
-						process.Locality["zoneid"] = zone
-					}
-				}
-			})
-
-			It("should report the coordinators as not valid", func() {
-				coordinatorStatus := make(map[string]bool, len(status.Client.Coordinators.Coordinators))
-				for _, coordinator := range status.Client.Coordinators.Coordinators {
-					coordinatorStatus[coordinator.Address.String()] = false
-				}
-
-				coordinatorsValid, addressesValid, err := checkCoordinatorValidity(cluster, status, coordinatorStatus)
-				Expect(coordinatorsValid).To(BeFalse())
-				Expect(addressesValid).To(BeTrue())
-				Expect(err).To(BeNil())
-			})
-		})
-
-		Context("with multiple regions", func() {
-			BeforeEach(func() {
-				coordinatorCount := 9
-				cluster.Spec.ProcessCounts.Storage = coordinatorCount
-
-				err = k8sClient.Update(context.TODO(), cluster)
-				Expect(err).NotTo(HaveOccurred())
-
-				result, err := reconcileCluster(cluster)
-				Expect(err).NotTo(HaveOccurred())
-				Expect(result.Requeue).To(BeFalse())
-
-				generation, err := reloadCluster(cluster)
-				Expect(err).NotTo(HaveOccurred())
-				Expect(generation).To(Equal(int64(2)))
-
-				status, err = adminClient.GetStatus()
-				Expect(err).NotTo(HaveOccurred())
-
-				cluster.Spec.DatabaseConfiguration.UsableRegions = 2
-
-				coordinators := make([]fdbv1beta2.FoundationDBStatusCoordinator, 0, coordinatorCount)
-				dc := 0
-				for _, process := range status.Cluster.Processes {
-					if process.ProcessClass == fdbv1beta2.ProcessClassStorage {
-						coordinators = append(coordinators, fdbv1beta2.FoundationDBStatusCoordinator{
-							Address:   process.Address,
-							Reachable: true,
-						})
-						dc++
-						process.Locality["dcid"] = fmt.Sprintf("dc%d", dc)
-						dc = dc % 3
-					}
-				}
-				status.Client.Coordinators.Coordinators = coordinators
-			})
-
-			Context("with coordinators divided across three DCs", func() {
-				It("should report the coordinators as valid", func() {
-					coordinatorStatus := make(map[string]bool, len(status.Client.Coordinators.Coordinators))
-					for _, coordinator := range status.Client.Coordinators.Coordinators {
-						coordinatorStatus[coordinator.Address.String()] = false
-					}
-
-					coordinatorsValid, addressesValid, err := checkCoordinatorValidity(cluster, status, coordinatorStatus)
-					Expect(coordinatorsValid).To(BeTrue())
-					Expect(addressesValid).To(BeTrue())
-					Expect(err).To(BeNil())
-				})
-			})
-
-			Context("with coordinators divided across two DCs", func() {
-				BeforeEach(func() {
-					for _, process := range status.Cluster.Processes {
-						if process.Locality["dcid"] == "dc3" {
-							process.Locality["dcid"] = "dc1"
-						}
-					}
-				})
-				It("should report the coordinators as not valid", func() {
-					coordinatorStatus := make(map[string]bool, len(status.Client.Coordinators.Coordinators))
-					for _, coordinator := range status.Client.Coordinators.Coordinators {
-						coordinatorStatus[coordinator.Address.String()] = false
-					}
-
-					coordinatorsValid, addressesValid, err := checkCoordinatorValidity(cluster, status, coordinatorStatus)
-					Expect(coordinatorsValid).To(BeFalse())
-					Expect(addressesValid).To(BeTrue())
-					Expect(err).To(BeNil())
-				})
-			})
-		})
-
-		When("changing the TLS setting", func() {
-			BeforeEach(func() {
-				cluster.Status.RequiredAddresses = fdbv1beta2.RequiredAddressSet{
-					TLS:    true,
-					NonTLS: true,
-				}
-			})
-
-			When("TLS is disabled", func() {
-				BeforeEach(func() {
-					cluster.Spec.MainContainer = fdbv1beta2.ContainerOverrides{
-						EnableTLS: false,
-					}
-					cluster.Spec.SidecarContainer = fdbv1beta2.ContainerOverrides{
-						EnableTLS: false,
-					}
-
-					adminClient, err = newMockAdminClient(cluster, k8sClient)
-					Expect(err).NotTo(HaveOccurred())
-
-					status, err = adminClient.GetStatus()
-					Expect(err).NotTo(HaveOccurred())
-				})
-
-				It("should report the coordinators addresses as valid", func() {
-					coordinatorStatus := make(map[string]bool, len(status.Client.Coordinators.Coordinators))
-					for _, coordinator := range status.Client.Coordinators.Coordinators {
-						coordinatorStatus[coordinator.Address.String()] = false
-					}
-
-					_, addressesValid, err := checkCoordinatorValidity(cluster, status, coordinatorStatus)
-					Expect(addressesValid).To(BeTrue())
-					Expect(err).To(BeNil())
-				})
-
-				When("converting back to TLS", func() {
-					BeforeEach(func() {
-						cluster.Spec.MainContainer = fdbv1beta2.ContainerOverrides{
-							EnableTLS: true,
-						}
-						cluster.Spec.SidecarContainer = fdbv1beta2.ContainerOverrides{
-							EnableTLS: true,
-						}
-
-						adminClient, err = newMockAdminClient(cluster, k8sClient)
-						Expect(err).NotTo(HaveOccurred())
-
-						status, err = adminClient.GetStatus()
-						Expect(err).NotTo(HaveOccurred())
-					})
-
-					It("should report the coordinators addresses as valid", func() {
-						coordinatorStatus := make(map[string]bool, len(status.Client.Coordinators.Coordinators))
-						for _, coordinator := range status.Client.Coordinators.Coordinators {
-							coordinatorStatus[coordinator.Address.String()] = false
-						}
-
-						_, addressesValid, err := checkCoordinatorValidity(cluster, status, coordinatorStatus)
-						Expect(addressesValid).To(BeTrue())
-						Expect(err).To(BeNil())
-					})
-				})
-			})
-
-			When("TLS is enabled", func() {
-				BeforeEach(func() {
-					cluster.Spec.MainContainer = fdbv1beta2.ContainerOverrides{
-						EnableTLS: true,
-					}
-					cluster.Spec.SidecarContainer = fdbv1beta2.ContainerOverrides{
-						EnableTLS: true,
-					}
-
-					adminClient, err = newMockAdminClient(cluster, k8sClient)
-					Expect(err).NotTo(HaveOccurred())
-
-					status, err = adminClient.GetStatus()
-					Expect(err).NotTo(HaveOccurred())
-				})
-
-				It("should report the coordinators addresses as valid", func() {
-					coordinatorStatus := make(map[string]bool, len(status.Client.Coordinators.Coordinators))
-					for _, coordinator := range status.Client.Coordinators.Coordinators {
-						coordinatorStatus[coordinator.Address.String()] = false
-					}
-
-					_, addressesValid, err := checkCoordinatorValidity(cluster, status, coordinatorStatus)
-					Expect(addressesValid).To(BeTrue())
-					Expect(err).To(BeNil())
-				})
-
-				When("converting back to non-TLS", func() {
-					BeforeEach(func() {
-						cluster.Spec.MainContainer = fdbv1beta2.ContainerOverrides{
-							EnableTLS: false,
-						}
-
-						cluster.Spec.SidecarContainer = fdbv1beta2.ContainerOverrides{
-							EnableTLS: false,
-						}
-
-						adminClient, err = newMockAdminClient(cluster, k8sClient)
-						Expect(err).NotTo(HaveOccurred())
-
-						status, err = adminClient.GetStatus()
-						Expect(err).NotTo(HaveOccurred())
-					})
-
-					It("should report the coordinators addresses as valid", func() {
-						coordinatorStatus := make(map[string]bool, len(status.Client.Coordinators.Coordinators))
-						for _, coordinator := range status.Client.Coordinators.Coordinators {
-							coordinatorStatus[coordinator.Address.String()] = false
-						}
-
-						_, addressesValid, err := checkCoordinatorValidity(cluster, status, coordinatorStatus)
-						Expect(addressesValid).To(BeTrue())
-						Expect(err).To(BeNil())
-					})
-				})
-			})
-		})
-
-		When("Generating the locality info based on a process", func() {
-			BeforeEach(func() {
-				cluster.Status.RequiredAddresses = fdbv1beta2.RequiredAddressSet{
-					TLS:    true,
-					NonTLS: true,
-				}
-
-				cluster.Spec.MainContainer = fdbv1beta2.ContainerOverrides{
-					EnableTLS: true,
-				}
-				cluster.Spec.SidecarContainer = fdbv1beta2.ContainerOverrides{
-					EnableTLS: true,
-				}
-
-				adminClient, err = newMockAdminClient(cluster, k8sClient)
-				Expect(err).NotTo(HaveOccurred())
-
-				status, err = adminClient.GetStatus()
-				Expect(err).NotTo(HaveOccurred())
-			})
-
-			When("TLS is disabled", func() {
-				It("the locality address should not contain tls", func() {
-					var proc fdbv1beta2.FoundationDBStatusProcessInfo
-					for _, p := range status.Cluster.Processes {
-						proc = p
-						// Select the first process Info
-						break
-					}
-
-					info, err := localityInfoForProcess(proc, false)
-					Expect(err).NotTo(HaveOccurred())
-					Expect(info.Address).NotTo(HaveSuffix("tls"))
-				})
-			})
-
-			When("TLS is enabled", func() {
-				It("the locality address should contain tls", func() {
-					var proc fdbv1beta2.FoundationDBStatusProcessInfo
-					for _, p := range status.Cluster.Processes {
-						proc = p
-						// Select the first process Info
-						break
-					}
-
-					info, err := localityInfoForProcess(proc, true)
-					Expect(err).NotTo(HaveOccurred())
-					Expect(info.Address).To(HaveSuffix("tls"))
-				})
-			})
-		})
-
-		When("enabling DNS names in the cluster file", func() {
-			BeforeEach(func() {
-				cluster.Spec.Routing.UseDNSInClusterFile = pointer.Bool(true)
-			})
-
-			Context("when the pods do not have DNS names assigned", func() {
-				It("should report valid coordinators", func() {
-					coordinatorStatus := make(map[string]bool, len(status.Client.Coordinators.Coordinators))
-					for _, coordinator := range status.Client.Coordinators.Coordinators {
-						coordinatorStatus[coordinator.Address.String()] = false
-					}
-
-					coordinatorsValid, addressesValid, err := checkCoordinatorValidity(cluster, status, coordinatorStatus)
-					Expect(err).NotTo(HaveOccurred())
-					Expect(coordinatorsValid).To(BeTrue())
-					Expect(addressesValid).To(BeTrue())
-				})
-			})
-
-			Context("when the pods have DNS names assigned", func() {
-				BeforeEach(func() {
-					pods := &corev1.PodList{}
-					err = k8sClient.List(context.TODO(), pods)
-					Expect(err).NotTo(HaveOccurred())
-
-					for _, pod := range pods.Items {
-						container := pod.Spec.Containers[1]
-						container.Env = append(container.Env, corev1.EnvVar{Name: "FDB_DNS_NAME", Value: internal.GetPodDNSName(cluster, pod.Name)})
-						pod.Spec.Containers[1] = container
-						err = k8sClient.Update(context.TODO(), &pod)
-						Expect(err).NotTo(HaveOccurred())
-					}
-
-					status, err = adminClient.GetStatus()
-					Expect(err).NotTo(HaveOccurred())
-				})
-
-				It("should reject coordinators based on IP addresses", func() {
-					coordinatorStatus := make(map[string]bool, len(status.Client.Coordinators.Coordinators))
-					for _, coordinator := range status.Client.Coordinators.Coordinators {
-						coordinatorStatus[coordinator.Address.String()] = false
-					}
-
-					coordinatorsValid, addressesValid, err := checkCoordinatorValidity(cluster, status, coordinatorStatus)
-					Expect(err).NotTo(HaveOccurred())
-					Expect(coordinatorsValid).To(BeFalse())
-					Expect(addressesValid).To(BeTrue())
-				})
-
-			})
-		})
-	})
-
 	Context("Setting the partial connection string", func() {
 		var partialConnectionString fdbv1beta2.ConnectionString
 
diff --git a/internal/locality/locality_test.go b/internal/locality/locality_test.go
new file mode 100644
index 00000000..f9741c9b
--- /dev/null
+++ b/internal/locality/locality_test.go
@@ -0,0 +1,1006 @@
+package locality
+
+/*
+ * change_coordinators_test.go
+ *
+ * This source file is part of the FoundationDB open source project
+ *
+ * Copyright 2022 Apple Inc. and the FoundationDB project authors
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import (
+	"fmt"
+	"math"
+	"net"
+	"strings"
+	"time"
+
+	"github.com/go-logr/logr"
+	corev1 "k8s.io/api/core/v1"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"k8s.io/utils/pointer"
+
+	fdbv1beta2 "github.com/FoundationDB/fdb-kubernetes-operator/api/v1beta2"
+	"github.com/FoundationDB/fdb-kubernetes-operator/internal"
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+)
+
+func generateDummyProcessInfo(id string, dcID string, port int, tls bool) fdbv1beta2.FoundationDBStatusProcessInfo {
+	ipString := "1.1.1." + strings.Split(id, "-")[1]
+
+	var tlsSuffix string
+	var flags map[string]bool
+	if tls {
+		flags = map[string]bool{"tls": true}
+		tlsSuffix = ":tls"
+	}
+
+	info := fdbv1beta2.FoundationDBStatusProcessInfo{
+		ProcessClass: fdbv1beta2.ProcessClassStorage,
+		Address: fdbv1beta2.ProcessAddress{
+			IPAddress: net.ParseIP(ipString),
+			Port:      port,
+			Flags:     flags,
+		},
+		CommandLine: fmt.Sprintf("... --public_address=%s:%d%s ...", ipString, port, tlsSuffix),
+		Locality: map[string]string{
+			fdbv1beta2.FDBLocalityInstanceIDKey: id,
+			fdbv1beta2.FDBLocalityZoneIDKey:     id,
+			fdbv1beta2.FDBLocalityDCIDKey:       dcID,
+		},
+	}
+
+	return info
+}
+
+func generateDefaultStatus(tls bool) *fdbv1beta2.FoundationDBStatus {
+	var flags map[string]bool
+
+	port := 4501
+	if tls {
+		port = 4500
+		flags = map[string]bool{"tls": true}
+	}
+
+	return &fdbv1beta2.FoundationDBStatus{
+		Client: fdbv1beta2.FoundationDBStatusLocalClientInfo{
+			Coordinators: fdbv1beta2.FoundationDBStatusCoordinatorInfo{
+				Coordinators: []fdbv1beta2.FoundationDBStatusCoordinator{
+					{
+						Address: fdbv1beta2.ProcessAddress{
+							IPAddress: net.ParseIP("1.1.1.1"),
+							Port:      port,
+							Flags:     flags,
+						},
+						Reachable: true,
+					},
+					{
+						Address: fdbv1beta2.ProcessAddress{
+							IPAddress: net.ParseIP("1.1.1.2"),
+							Port:      port,
+							Flags:     flags,
+						},
+						Reachable: true,
+					},
+					{
+						Address: fdbv1beta2.ProcessAddress{
+							IPAddress: net.ParseIP("1.1.1.3"),
+							Port:      port,
+							Flags:     flags,
+						},
+						Reachable: true,
+					},
+				},
+			},
+		},
+		Cluster: fdbv1beta2.FoundationDBStatusClusterInfo{
+			Processes: map[string]fdbv1beta2.FoundationDBStatusProcessInfo{
+				"1": generateDummyProcessInfo("test-1", "dc1", port, tls),
+				"2": generateDummyProcessInfo("test-2", "dc1", port, tls),
+				"3": generateDummyProcessInfo("test-3", "dc1", port, tls),
+			},
+		},
+	}
+}
+
+var _ = Describe("Change coordinators", func() {
+	var cluster *fdbv1beta2.FoundationDBCluster
+
+	BeforeEach(func() {
+		cluster = internal.CreateDefaultCluster()
+		disabled := false
+		cluster.Spec.LockOptions.DisableLocks = &disabled
+		cluster.Spec.CoordinatorSelection = []fdbv1beta2.CoordinatorSelectionSetting{
+			{
+				ProcessClass: fdbv1beta2.ProcessClassStorage,
+				Priority:     math.MaxInt32,
+			},
+			{
+				ProcessClass: fdbv1beta2.ProcessClassLog,
+				Priority:     0,
+			},
+		}
+	})
+
+	When("Sorting the localities", func() {
+		var localities []Info
+
+		BeforeEach(func() {
+			localities = []Info{
+				{
+					ID:    "storage-1",
+					Class: fdbv1beta2.ProcessClassStorage,
+				},
+				{
+					ID:    "tlog-1",
+					Class: fdbv1beta2.ProcessClassTransaction,
+				},
+				{
+					ID:    "log-1",
+					Class: fdbv1beta2.ProcessClassLog,
+				},
+				{
+					ID:    "storage-51",
+					Class: fdbv1beta2.ProcessClassStorage,
+				},
+			}
+		})
+
+		When("no other preferences are defined", func() {
+			BeforeEach(func() {
+				cluster.Spec.CoordinatorSelection = []fdbv1beta2.CoordinatorSelectionSetting{}
+			})
+
+			It("should sort the localities based on the IDs", func() {
+				sortLocalities(cluster, localities)
+
+				Expect(localities[0].Class).To(Equal(fdbv1beta2.ProcessClassLog))
+				Expect(localities[0].ID).To(Equal("log-1"))
+				Expect(localities[1].Class).To(Equal(fdbv1beta2.ProcessClassStorage))
+				Expect(localities[1].ID).To(Equal("storage-1"))
+				Expect(localities[2].Class).To(Equal(fdbv1beta2.ProcessClassStorage))
+				Expect(localities[2].ID).To(Equal("storage-51"))
+				Expect(localities[3].Class).To(Equal(fdbv1beta2.ProcessClassTransaction))
+				Expect(localities[3].ID).To(Equal("tlog-1"))
+			})
+		})
+
+		When("when the storage class is preferred", func() {
+			BeforeEach(func() {
+				cluster.Spec.CoordinatorSelection = []fdbv1beta2.CoordinatorSelectionSetting{
+					{
+						ProcessClass: fdbv1beta2.ProcessClassStorage,
+						Priority:     0,
+					},
+				}
+			})
+
+			It("should sort the localities based on the provided config", func() {
+				sortLocalities(cluster, localities)
+
+				Expect(localities[0].Class).To(Equal(fdbv1beta2.ProcessClassStorage))
+				Expect(localities[0].ID).To(Equal("storage-1"))
+				Expect(localities[1].Class).To(Equal(fdbv1beta2.ProcessClassStorage))
+				Expect(localities[1].ID).To(Equal("storage-51"))
+				Expect(localities[2].Class).To(Equal(fdbv1beta2.ProcessClassLog))
+				Expect(localities[2].ID).To(Equal("log-1"))
+				Expect(localities[3].Class).To(Equal(fdbv1beta2.ProcessClassTransaction))
+				Expect(localities[3].ID).To(Equal("tlog-1"))
+			})
+		})
+
+		When("when the storage class is preferred over transaction class", func() {
+			BeforeEach(func() {
+				cluster.Spec.CoordinatorSelection = []fdbv1beta2.CoordinatorSelectionSetting{
+					{
+						ProcessClass: fdbv1beta2.ProcessClassStorage,
+						Priority:     1,
+					},
+					{
+						ProcessClass: fdbv1beta2.ProcessClassTransaction,
+						Priority:     0,
+					},
+				}
+			})
+
+			It("should sort the localities based on the provided config", func() {
+				sortLocalities(cluster, localities)
+
+				Expect(localities[0].Class).To(Equal(fdbv1beta2.ProcessClassStorage))
+				Expect(localities[0].ID).To(Equal("storage-1"))
+				Expect(localities[1].Class).To(Equal(fdbv1beta2.ProcessClassStorage))
+				Expect(localities[1].ID).To(Equal("storage-51"))
+				Expect(localities[2].Class).To(Equal(fdbv1beta2.ProcessClassTransaction))
+				Expect(localities[2].ID).To(Equal("tlog-1"))
+				Expect(localities[3].Class).To(Equal(fdbv1beta2.ProcessClassLog))
+				Expect(localities[3].ID).To(Equal("log-1"))
+			})
+		})
+	})
+
+	Describe("chooseDistributedProcesses", func() {
+		var candidates []Info
+		var result []Info
+		var err error
+
+		Context("with a flat set of processes", func() {
+			BeforeEach(func() {
+				candidates = []Info{
+					{ID: "p1", LocalityData: map[string]string{fdbv1beta2.FDBLocalityZoneIDKey: "z1"}},
+					{ID: "p2", LocalityData: map[string]string{fdbv1beta2.FDBLocalityZoneIDKey: "z1"}},
+					{ID: "p3", LocalityData: map[string]string{fdbv1beta2.FDBLocalityZoneIDKey: "z2"}},
+					{ID: "p4", LocalityData: map[string]string{fdbv1beta2.FDBLocalityZoneIDKey: "z3"}},
+					{ID: "p5", LocalityData: map[string]string{fdbv1beta2.FDBLocalityZoneIDKey: "z2"}},
+					{ID: "p6", LocalityData: map[string]string{fdbv1beta2.FDBLocalityZoneIDKey: "z4"}},
+					{ID: "p7", LocalityData: map[string]string{fdbv1beta2.FDBLocalityZoneIDKey: "z5"}},
+				}
+				result, err = ChooseDistributedProcesses(cluster, candidates, 5, ProcessSelectionConstraint{})
+				Expect(err).NotTo(HaveOccurred())
+			})
+
+			It("should recruit the processes across multiple zones", func() {
+				Expect(len(result)).To(Equal(5))
+				Expect(result[0].ID).To(Equal("p1"))
+				Expect(result[1].ID).To(Equal("p3"))
+				Expect(result[2].ID).To(Equal("p4"))
+				Expect(result[3].ID).To(Equal("p6"))
+				Expect(result[4].ID).To(Equal("p7"))
+			})
+		})
+
+		Context("with fewer zones than desired processes", func() {
+			BeforeEach(func() {
+				candidates = []Info{
+					{ID: "p1", LocalityData: map[string]string{fdbv1beta2.FDBLocalityZoneIDKey: "z1"}},
+					{ID: "p2", LocalityData: map[string]string{fdbv1beta2.FDBLocalityZoneIDKey: "z1"}},
+					{ID: "p3", LocalityData: map[string]string{fdbv1beta2.FDBLocalityZoneIDKey: "z2"}},
+					{ID: "p4", LocalityData: map[string]string{fdbv1beta2.FDBLocalityZoneIDKey: "z3"}},
+					{ID: "p5", LocalityData: map[string]string{fdbv1beta2.FDBLocalityZoneIDKey: "z2"}},
+					{ID: "p6", LocalityData: map[string]string{fdbv1beta2.FDBLocalityZoneIDKey: "z4"}},
+				}
+			})
+
+			Context("with no hard limit", func() {
+				It("should only re-use zones as necessary", func() {
+					result, err = ChooseDistributedProcesses(cluster, candidates, 5, ProcessSelectionConstraint{})
+					Expect(err).NotTo(HaveOccurred())
+
+					Expect(len(result)).To(Equal(5))
+					Expect(result[0].ID).To(Equal("p1"))
+					Expect(result[1].ID).To(Equal("p3"))
+					Expect(result[2].ID).To(Equal("p4"))
+					Expect(result[3].ID).To(Equal("p6"))
+					Expect(result[4].ID).To(Equal("p2"))
+				})
+			})
+
+			Context("with a hard limit", func() {
+				It("should give an error", func() {
+					result, err = ChooseDistributedProcesses(cluster, candidates, 5, ProcessSelectionConstraint{
+						HardLimits: map[string]int{fdbv1beta2.FDBLocalityZoneIDKey: 1},
+					})
+					Expect(err).To(HaveOccurred())
+					Expect(err.Error()).To(Equal("Could only select 4 processes, but 5 are required"))
+				})
+			})
+		})
+
+		Context("with multiple data centers", func() {
+			BeforeEach(func() {
+				candidates = []Info{
+					{ID: "p1", LocalityData: map[string]string{fdbv1beta2.FDBLocalityZoneIDKey: "z1", "dcid": "dc1"}},
+					{ID: "p2", LocalityData: map[string]string{fdbv1beta2.FDBLocalityZoneIDKey: "z1", "dcid": "dc1"}},
+					{ID: "p3", LocalityData: map[string]string{fdbv1beta2.FDBLocalityZoneIDKey: "z2", "dcid": "dc1"}},
+					{ID: "p4", LocalityData: map[string]string{fdbv1beta2.FDBLocalityZoneIDKey: "z3", "dcid": "dc1"}},
+					{ID: "p5", LocalityData: map[string]string{fdbv1beta2.FDBLocalityZoneIDKey: "z2", "dcid": "dc1"}},
+					{ID: "p6", LocalityData: map[string]string{fdbv1beta2.FDBLocalityZoneIDKey: "z4", "dcid": "dc1"}},
+					{ID: "p7", LocalityData: map[string]string{fdbv1beta2.FDBLocalityZoneIDKey: "z5", "dcid": "dc1"}},
+					{ID: "p8", LocalityData: map[string]string{fdbv1beta2.FDBLocalityZoneIDKey: "z6", "dcid": "dc2"}},
+					{ID: "p9", LocalityData: map[string]string{fdbv1beta2.FDBLocalityZoneIDKey: "z7", "dcid": "dc2"}},
+					{ID: "p10", LocalityData: map[string]string{fdbv1beta2.FDBLocalityZoneIDKey: "z8", "dcid": "dc2"}},
+				}
+			})
+
+			Context("with the default constraints", func() {
+				BeforeEach(func() {
+					result, err = ChooseDistributedProcesses(cluster, candidates, 5, ProcessSelectionConstraint{})
+					Expect(err).NotTo(HaveOccurred())
+				})
+
+				It("should recruit the processes across multiple zones and data centers", func() {
+					Expect(len(result)).To(Equal(5))
+					Expect(result[0].ID).To(Equal("p1"))
+					Expect(result[1].ID).To(Equal("p10"))
+					Expect(result[2].ID).To(Equal("p3"))
+					Expect(result[3].ID).To(Equal("p8"))
+					Expect(result[4].ID).To(Equal("p4"))
+				})
+			})
+
+			Context("when only distributing across data centers", func() {
+				BeforeEach(func() {
+					result, err = ChooseDistributedProcesses(cluster, candidates, 5, ProcessSelectionConstraint{
+						Fields: []string{"dcid"},
+					})
+					Expect(err).NotTo(HaveOccurred())
+				})
+
+				It("should recruit the processes across data centers", func() {
+					Expect(len(result)).To(Equal(5))
+					Expect(result[0].ID).To(Equal("p1"))
+					Expect(result[1].ID).To(Equal("p10"))
+					Expect(result[2].ID).To(Equal("p2"))
+					Expect(result[3].ID).To(Equal("p8"))
+					Expect(result[4].ID).To(Equal("p3"))
+				})
+			})
+		})
+	})
+
+	DescribeTable("when getting the hard limits", func(cluster *fdbv1beta2.FoundationDBCluster, expected map[string]int) {
+		Expect(GetHardLimits(cluster)).To(Equal(expected))
+	},
+		Entry("default cluster with one usable region",
+			&fdbv1beta2.FoundationDBCluster{},
+			map[string]int{
+				fdbv1beta2.FDBLocalityZoneIDKey: 1,
+			},
+		),
+		Entry("default cluster with two usable regiosn",
+			&fdbv1beta2.FoundationDBCluster{
+				Spec: fdbv1beta2.FoundationDBClusterSpec{
+					DatabaseConfiguration: fdbv1beta2.DatabaseConfiguration{
+						UsableRegions: 2,
+					},
+				},
+			},
+			map[string]int{
+				fdbv1beta2.FDBLocalityZoneIDKey: 1,
+				fdbv1beta2.FDBLocalityDCIDKey:   4,
+			},
+		),
+	)
+
+	DescribeTable("when getting the locality info from a process", func(process fdbv1beta2.FoundationDBStatusProcessInfo, mainContainerTLS bool, expected Info, expectedError bool) {
+		info, err := InfoForProcess(process, mainContainerTLS)
+		if expectedError {
+			Expect(err).To(HaveOccurred())
+			return
+		}
+
+		Expect(err).NotTo(HaveOccurred())
+		Expect(info).To(Equal(expected))
+	},
+		Entry("a process info without tls",
+			fdbv1beta2.FoundationDBStatusProcessInfo{
+				CommandLine: "... --public_address=1.1.1.1:4501,1.1.1.1:4500:tls ...",
+				Locality: map[string]string{
+					fdbv1beta2.FDBLocalityInstanceIDKey: "test",
+				},
+				ProcessClass: fdbv1beta2.ProcessClassStorage,
+			},
+			false,
+			Info{
+				ID: "test",
+				Address: fdbv1beta2.ProcessAddress{
+					IPAddress: net.ParseIP("1.1.1.1"),
+					Port:      4501,
+				},
+				Class: fdbv1beta2.ProcessClassStorage,
+				LocalityData: map[string]string{
+					fdbv1beta2.FDBLocalityInstanceIDKey: "test",
+				},
+			},
+			false,
+		),
+		Entry("a process info with tls",
+			fdbv1beta2.FoundationDBStatusProcessInfo{
+				CommandLine: "... --public_address=1.1.1.1:4501,1.1.1.1:4500:tls ...",
+				Locality: map[string]string{
+					fdbv1beta2.FDBLocalityInstanceIDKey: "test",
+				},
+				ProcessClass: fdbv1beta2.ProcessClassStorage,
+			},
+			true,
+			Info{
+				ID: "test",
+				Address: fdbv1beta2.ProcessAddress{
+					IPAddress: net.ParseIP("1.1.1.1"),
+					Port:      4500,
+					Flags: map[string]bool{
+						"tls": true,
+					},
+				},
+				Class: fdbv1beta2.ProcessClassStorage,
+				LocalityData: map[string]string{
+					fdbv1beta2.FDBLocalityInstanceIDKey: "test",
+				},
+			},
+			false,
+		),
+		Entry("a process info with missing public address",
+			fdbv1beta2.FoundationDBStatusProcessInfo{
+				CommandLine: "",
+				Locality: map[string]string{
+					fdbv1beta2.FDBLocalityInstanceIDKey: "test",
+				},
+				ProcessClass: fdbv1beta2.ProcessClassStorage,
+			},
+			true,
+			Info{
+				ID: "test",
+				Address: fdbv1beta2.ProcessAddress{
+					IPAddress: net.ParseIP("1.1.1.1"),
+					Port:      4500,
+					Flags: map[string]bool{
+						"tls": true,
+					},
+				},
+				Class: fdbv1beta2.ProcessClassStorage,
+				LocalityData: map[string]string{
+					fdbv1beta2.FDBLocalityInstanceIDKey: "test",
+				},
+			},
+			true,
+		),
+	)
+
+	DescribeTable("when getting the locality info from a sidecar", func(cluster *fdbv1beta2.FoundationDBCluster, pod *corev1.Pod, expected Info, expectedError bool) {
+		client, err := internal.NewMockFdbPodClient(cluster, pod)
+		Expect(err).NotTo(HaveOccurred())
+
+		info, err := InfoFromSidecar(cluster, client)
+		if expectedError {
+			Expect(err).To(HaveOccurred())
+			return
+		}
+
+		Expect(err).NotTo(HaveOccurred())
+		Expect(info).To(Equal(expected))
+	},
+		Entry("a Pod with all fields set",
+			&fdbv1beta2.FoundationDBCluster{
+				Spec: fdbv1beta2.FoundationDBClusterSpec{
+					FaultDomain: fdbv1beta2.FoundationDBClusterFaultDomain{
+						Key: "foundationdb.org/none",
+					},
+				},
+				Status: fdbv1beta2.FoundationDBClusterStatus{
+					RequiredAddresses: fdbv1beta2.RequiredAddressSet{
+						NonTLS: true,
+					},
+				},
+			},
+			&corev1.Pod{
+				ObjectMeta: metav1.ObjectMeta{
+					Name: "test",
+					Labels: map[string]string{
+						fdbv1beta2.FDBProcessGroupIDLabel: "test",
+					},
+				},
+				Spec: corev1.PodSpec{
+					Containers: []corev1.Container{
+						{
+							Name: "foundationdb-kubernetes-sidecar",
+							Args: []string{
+								"--public-ip-family",
+								"4",
+							},
+						},
+					},
+				},
+				Status: corev1.PodStatus{
+					PodIPs: []corev1.PodIP{
+						{IP: "1.1.1.1"},
+					},
+				},
+			},
+			Info{
+				ID: "test",
+				Address: fdbv1beta2.ProcessAddress{
+					IPAddress: net.ParseIP("1.1.1.1"),
+					Port:      4501,
+				},
+				LocalityData: map[string]string{
+					fdbv1beta2.FDBLocalityZoneIDKey:  "test",
+					fdbv1beta2.FDBLocalityDNSNameKey: "",
+				},
+			},
+			false,
+		),
+		Entry("the sidecar is not reachable",
+			&fdbv1beta2.FoundationDBCluster{
+				Spec: fdbv1beta2.FoundationDBClusterSpec{
+					FaultDomain: fdbv1beta2.FoundationDBClusterFaultDomain{
+						Key: "foundationdb.org/none",
+					},
+				},
+				Status: fdbv1beta2.FoundationDBClusterStatus{
+					RequiredAddresses: fdbv1beta2.RequiredAddressSet{
+						NonTLS: true,
+					},
+				},
+			},
+			&corev1.Pod{
+				ObjectMeta: metav1.ObjectMeta{
+					Annotations: map[string]string{
+						internal.MockUnreachableAnnotation: "true",
+					},
+				},
+			},
+			Info{},
+			true,
+		),
+	)
+
+	Describe("checkCoordinatorValidity", func() {
+		var status *fdbv1beta2.FoundationDBStatus
+		var coordinatorStatus map[string]bool
+
+		BeforeEach(func() {
+			cluster.Status.ProcessGroups = []*fdbv1beta2.ProcessGroupStatus{
+				{
+					ProcessGroupID: "test-1",
+				},
+				{
+					ProcessGroupID: "test-2",
+				},
+				{
+					ProcessGroupID: "test-3",
+				},
+			}
+
+			status = generateDefaultStatus(false)
+		})
+
+		JustBeforeEach(func() {
+			coordinatorStatus = make(map[string]bool, len(status.Client.Coordinators.Coordinators))
+			for _, coordinator := range status.Client.Coordinators.Coordinators {
+				coordinatorStatus[coordinator.Address.String()] = false
+			}
+		})
+
+		Context("with the default configuration", func() {
+			It("should report the coordinators as valid", func() {
+				coordinatorsValid, addressesValid, err := CheckCoordinatorValidity(logr.Discard(), cluster, status, coordinatorStatus)
+				Expect(coordinatorsValid).To(BeTrue())
+				Expect(addressesValid).To(BeTrue())
+				Expect(err).NotTo(HaveOccurred())
+			})
+		})
+
+		When("an empty coordinator status is passed down", func() {
+			It("should report the coordinators as invalid", func() {
+				coordinatorsValid, addressesValid, err := CheckCoordinatorValidity(logr.Discard(), cluster, status, nil)
+				Expect(coordinatorsValid).To(BeFalse())
+				Expect(addressesValid).To(BeFalse())
+				Expect(err).To(HaveOccurred())
+			})
+		})
+
+		When("changing the coordinator selection setting", func() {
+			BeforeEach(func() {
+				cluster.Spec.CoordinatorSelection = []fdbv1beta2.CoordinatorSelectionSetting{
+					{
+						ProcessClass: fdbv1beta2.ProcessClassLog,
+					},
+				}
+			})
+
+			It("should report the coordinators as invalid", func() {
+				coordinatorsValid, addressesValid, err := CheckCoordinatorValidity(logr.Discard(), cluster, status, coordinatorStatus)
+				Expect(coordinatorsValid).To(BeFalse())
+				Expect(addressesValid).To(BeTrue())
+				Expect(err).To(BeNil())
+			})
+		})
+
+		When("one process is marked for exclusion", func() {
+			BeforeEach(func() {
+				process := status.Cluster.Processes["1"]
+				process.Excluded = true
+				status.Cluster.Processes["1"] = process
+			})
+
+			It("should report the coordinators as invalid", func() {
+				coordinatorsValid, addressesValid, err := CheckCoordinatorValidity(logr.Discard(), cluster, status, coordinatorStatus)
+				Expect(coordinatorsValid).To(BeFalse())
+				Expect(addressesValid).To(BeTrue())
+				Expect(err).NotTo(HaveOccurred())
+			})
+		})
+
+		When("a process has an empty address", func() {
+			BeforeEach(func() {
+				status.Cluster.Processes["4"] = fdbv1beta2.FoundationDBStatusProcessInfo{}
+			})
+
+			It("should ignore the process", func() {
+				coordinatorsValid, addressesValid, err := CheckCoordinatorValidity(logr.Discard(), cluster, status, coordinatorStatus)
+				Expect(coordinatorsValid).To(BeTrue())
+				Expect(addressesValid).To(BeTrue())
+				Expect(err).To(BeNil())
+			})
+		})
+
+		When("the coordinators are listening on TLS", func() {
+			BeforeEach(func() {
+				cluster.Spec.MainContainer.EnableTLS = true
+			})
+
+			It("should report the coordinators as invalid", func() {
+				coordinatorsValid, addressesValid, err := CheckCoordinatorValidity(logr.Discard(), cluster, status, coordinatorStatus)
+				Expect(coordinatorsValid).To(BeFalse())
+				Expect(addressesValid).To(BeFalse())
+				Expect(err).NotTo(HaveOccurred())
+			})
+		})
+
+		When("a process has a process group status with a pending Pod", func() {
+			BeforeEach(func() {
+				status.Cluster.Processes["4"] = fdbv1beta2.FoundationDBStatusProcessInfo{
+					Address: fdbv1beta2.ProcessAddress{
+						IPAddress: net.ParseIP("1.1.1.4"),
+						Port:      4501,
+					},
+					Locality: map[string]string{
+						fdbv1beta2.FDBLocalityInstanceIDKey: "test-4",
+					},
+				}
+
+				cluster.Status.ProcessGroups = append(cluster.Status.ProcessGroups,
+					&fdbv1beta2.ProcessGroupStatus{
+						ProcessGroupID: "test-4",
+						ProcessGroupConditions: []*fdbv1beta2.ProcessGroupCondition{
+							{
+								ProcessGroupConditionType: fdbv1beta2.PodPending,
+								Timestamp:                 time.Now().Add(-45 * time.Minute).Unix(),
+							},
+						},
+					})
+			})
+
+			It("should ignore the process"+
+				"", func() {
+				coordinatorsValid, addressesValid, err := CheckCoordinatorValidity(logr.Discard(), cluster, status, coordinatorStatus)
+				Expect(coordinatorsValid).To(BeTrue())
+				Expect(addressesValid).To(BeTrue())
+				Expect(err).To(BeNil())
+			})
+		})
+
+		When("a process misses the public-address flag", func() {
+			BeforeEach(func() {
+				status.Cluster.Processes["4"] = fdbv1beta2.FoundationDBStatusProcessInfo{
+					Address: fdbv1beta2.ProcessAddress{
+						IPAddress: net.ParseIP("1.1.1.4"),
+						Port:      4501,
+					},
+				}
+			})
+
+			It("should report that not all addresses are valid", func() {
+				coordinatorsValid, addressesValid, err := CheckCoordinatorValidity(logr.Discard(), cluster, status, coordinatorStatus)
+				Expect(coordinatorsValid).To(BeTrue())
+				Expect(addressesValid).To(BeFalse())
+				Expect(err).To(BeNil())
+			})
+		})
+
+		Context("with a tester process without a command line field", func() {
+			BeforeEach(func() {
+				testProcess := fdbv1beta2.FoundationDBStatusProcessInfo{
+					Address: fdbv1beta2.ProcessAddress{
+						IPAddress: net.ParseIP("9.9.9.9"),
+					},
+					ProcessClass: fdbv1beta2.ProcessClassTest,
+				}
+				Expect(testProcess.Address.IsEmpty()).To(BeFalse())
+				status.Cluster.Processes[string(fdbv1beta2.ProcessClassTest)] = testProcess
+			})
+
+			It("should ignore the test process and report the coordinators as valid", func() {
+				coordinatorsValid, addressesValid, err := CheckCoordinatorValidity(logr.Discard(), cluster, status, coordinatorStatus)
+				Expect(coordinatorsValid).To(BeTrue())
+				Expect(addressesValid).To(BeTrue())
+				Expect(err).To(BeNil())
+			})
+		})
+
+		Context("with too few coordinators", func() {
+			BeforeEach(func() {
+				status.Client.Coordinators.Coordinators = status.Client.Coordinators.Coordinators[0:2]
+			})
+
+			It("should report the coordinators as not valid", func() {
+				coordinatorsValid, addressesValid, err := CheckCoordinatorValidity(logr.Discard(), cluster, status, coordinatorStatus)
+				Expect(coordinatorsValid).To(BeFalse())
+				Expect(addressesValid).To(BeTrue())
+				Expect(err).To(BeNil())
+			})
+		})
+
+		Context("with too few zones", func() {
+			BeforeEach(func() {
+				zone := ""
+				for _, process := range status.Cluster.Processes {
+					if process.Address.Equal(status.Client.Coordinators.Coordinators[0].Address) {
+						zone = process.Locality[fdbv1beta2.FDBLocalityZoneIDKey]
+					}
+				}
+				for _, process := range status.Cluster.Processes {
+					if process.Address.Equal(status.Client.Coordinators.Coordinators[1].Address) {
+						process.Locality[fdbv1beta2.FDBLocalityZoneIDKey] = zone
+					}
+				}
+			})
+
+			It("should report the coordinators as not valid", func() {
+				coordinatorsValid, addressesValid, err := CheckCoordinatorValidity(logr.Discard(), cluster, status, coordinatorStatus)
+				Expect(coordinatorsValid).To(BeFalse())
+				Expect(addressesValid).To(BeTrue())
+				Expect(err).To(BeNil())
+			})
+		})
+
+		Context("with multiple regions", func() {
+			BeforeEach(func() {
+				cluster.Spec.DatabaseConfiguration.UsableRegions = 2
+
+				status.Cluster.Processes["4"] = generateDummyProcessInfo("test-4", "dc2", 4501, false)
+				status.Cluster.Processes["5"] = generateDummyProcessInfo("test-5", "dc2", 4501, false)
+				status.Cluster.Processes["6"] = generateDummyProcessInfo("test-6", "dc2", 4501, false)
+				status.Cluster.Processes["7"] = generateDummyProcessInfo("test-7", "dc3", 4501, false)
+				status.Cluster.Processes["8"] = generateDummyProcessInfo("test-8", "dc3", 4501, false)
+				status.Cluster.Processes["9"] = generateDummyProcessInfo("test-9", "dc3", 4501, false)
+
+				status.Client.Coordinators.Coordinators = append(status.Client.Coordinators.Coordinators,
+					fdbv1beta2.FoundationDBStatusCoordinator{
+						Address: fdbv1beta2.ProcessAddress{
+							IPAddress: net.ParseIP("1.1.1.4"),
+							Port:      4501,
+						},
+						Reachable: true,
+					},
+					fdbv1beta2.FoundationDBStatusCoordinator{
+						Address: fdbv1beta2.ProcessAddress{
+							IPAddress: net.ParseIP("1.1.1.5"),
+							Port:      4501,
+						},
+						Reachable: true,
+					},
+					fdbv1beta2.FoundationDBStatusCoordinator{
+						Address: fdbv1beta2.ProcessAddress{
+							IPAddress: net.ParseIP("1.1.1.6"),
+							Port:      4501,
+						},
+						Reachable: true,
+					},
+					fdbv1beta2.FoundationDBStatusCoordinator{
+						Address: fdbv1beta2.ProcessAddress{
+							IPAddress: net.ParseIP("1.1.1.7"),
+							Port:      4501,
+						},
+						Reachable: true,
+					},
+					fdbv1beta2.FoundationDBStatusCoordinator{
+						Address: fdbv1beta2.ProcessAddress{
+							IPAddress: net.ParseIP("1.1.1.8"),
+							Port:      4501,
+						},
+						Reachable: true,
+					},
+					fdbv1beta2.FoundationDBStatusCoordinator{
+						Address: fdbv1beta2.ProcessAddress{
+							IPAddress: net.ParseIP("1.1.1.9"),
+							Port:      4501,
+						},
+						Reachable: true,
+					},
+				)
+			})
+
+			Context("with coordinators divided across three DCs", func() {
+				It("should report the coordinators as valid", func() {
+					coordinatorsValid, addressesValid, err := CheckCoordinatorValidity(logr.Discard(), cluster, status, coordinatorStatus)
+					Expect(coordinatorsValid).To(BeTrue())
+					Expect(addressesValid).To(BeTrue())
+					Expect(err).To(BeNil())
+				})
+			})
+
+			Context("with coordinators divided across two DCs", func() {
+				BeforeEach(func() {
+					for _, process := range status.Cluster.Processes {
+						if process.Locality["dcid"] == "dc3" {
+							process.Locality["dcid"] = "dc1"
+						}
+					}
+				})
+				It("should report the coordinators as not valid", func() {
+					coordinatorsValid, addressesValid, err := CheckCoordinatorValidity(logr.Discard(), cluster, status, coordinatorStatus)
+					Expect(coordinatorsValid).To(BeFalse())
+					Expect(addressesValid).To(BeTrue())
+					Expect(err).To(BeNil())
+				})
+			})
+		})
+
+		When("changing the TLS setting", func() {
+			BeforeEach(func() {
+				cluster.Status.RequiredAddresses = fdbv1beta2.RequiredAddressSet{
+					TLS:    true,
+					NonTLS: true,
+				}
+			})
+
+			When("TLS is disabled", func() {
+				BeforeEach(func() {
+					cluster.Spec.MainContainer = fdbv1beta2.ContainerOverrides{
+						EnableTLS: false,
+					}
+					cluster.Spec.SidecarContainer = fdbv1beta2.ContainerOverrides{
+						EnableTLS: false,
+					}
+				})
+
+				It("should report the coordinators addresses as valid", func() {
+					_, addressesValid, err := CheckCoordinatorValidity(logr.Discard(), cluster, status, coordinatorStatus)
+					Expect(addressesValid).To(BeTrue())
+					Expect(err).To(BeNil())
+				})
+
+				When("converting back to TLS", func() {
+					BeforeEach(func() {
+						cluster.Spec.MainContainer = fdbv1beta2.ContainerOverrides{
+							EnableTLS: true,
+						}
+						cluster.Spec.SidecarContainer = fdbv1beta2.ContainerOverrides{
+							EnableTLS: true,
+						}
+
+						status = generateDefaultStatus(true)
+					})
+
+					It("should report the coordinators addresses as valid", func() {
+						_, addressesValid, err := CheckCoordinatorValidity(logr.Discard(), cluster, status, coordinatorStatus)
+						Expect(addressesValid).To(BeTrue())
+						Expect(err).To(BeNil())
+					})
+				})
+			})
+
+			When("TLS is enabled", func() {
+				BeforeEach(func() {
+					cluster.Spec.MainContainer = fdbv1beta2.ContainerOverrides{
+						EnableTLS: true,
+					}
+					cluster.Spec.SidecarContainer = fdbv1beta2.ContainerOverrides{
+						EnableTLS: true,
+					}
+
+					status = generateDefaultStatus(true)
+				})
+
+				It("should report the coordinators addresses as valid", func() {
+					_, addressesValid, err := CheckCoordinatorValidity(logr.Discard(), cluster, status, coordinatorStatus)
+					Expect(addressesValid).To(BeTrue())
+					Expect(err).To(BeNil())
+				})
+
+				When("converting back to non-TLS", func() {
+					BeforeEach(func() {
+						cluster.Spec.MainContainer = fdbv1beta2.ContainerOverrides{
+							EnableTLS: false,
+						}
+
+						cluster.Spec.SidecarContainer = fdbv1beta2.ContainerOverrides{
+							EnableTLS: false,
+						}
+
+						status = generateDefaultStatus(false)
+					})
+
+					It("should report the coordinators addresses as valid", func() {
+						_, addressesValid, err := CheckCoordinatorValidity(logr.Discard(), cluster, status, coordinatorStatus)
+						Expect(addressesValid).To(BeTrue())
+						Expect(err).To(BeNil())
+					})
+				})
+			})
+		})
+
+		When("enabling DNS names in the cluster file", func() {
+			BeforeEach(func() {
+				cluster.Spec.Routing.UseDNSInClusterFile = pointer.Bool(true)
+			})
+
+			When("the pods do not have DNS names assigned", func() {
+				It("should report valid coordinators", func() {
+					coordinatorsValid, addressesValid, err := CheckCoordinatorValidity(logr.Discard(), cluster, status, coordinatorStatus)
+					Expect(err).NotTo(HaveOccurred())
+					Expect(coordinatorsValid).To(BeTrue())
+					Expect(addressesValid).To(BeTrue())
+				})
+			})
+
+			When("the pods have DNS names assigned", func() {
+				BeforeEach(func() {
+					for _, process := range status.Cluster.Processes {
+						process.Locality[fdbv1beta2.FDBLocalityDNSNameKey] = process.Locality[fdbv1beta2.FDBLocalityZoneIDKey]
+					}
+				})
+
+				It("should reject coordinators based on IP addresses", func() {
+					coordinatorsValid, addressesValid, err := CheckCoordinatorValidity(logr.Discard(), cluster, status, coordinatorStatus)
+					Expect(err).NotTo(HaveOccurred())
+					Expect(coordinatorsValid).To(BeFalse())
+					Expect(addressesValid).To(BeTrue())
+				})
+			})
+
+			When("the pods have DNS names assigned and the coordinator have DNS names", func() {
+				BeforeEach(func() {
+					status.Client.Coordinators.Coordinators = nil
+
+					for _, process := range status.Cluster.Processes {
+						process.Locality[fdbv1beta2.FDBLocalityDNSNameKey] = process.Locality[fdbv1beta2.FDBLocalityZoneIDKey]
+						status.Client.Coordinators.Coordinators = append(status.Client.Coordinators.Coordinators,
+							fdbv1beta2.FoundationDBStatusCoordinator{
+								Address: fdbv1beta2.ProcessAddress{
+									StringAddress: process.Locality[fdbv1beta2.FDBLocalityZoneIDKey],
+									Port:          4501,
+								},
+							})
+					}
+				})
+
+				It("should return that the coordinators are valid", func() {
+					coordinatorsValid, addressesValid, err := CheckCoordinatorValidity(logr.Discard(), cluster, status, coordinatorStatus)
+					Expect(coordinatorsValid).To(BeTrue())
+					Expect(addressesValid).To(BeTrue())
+					Expect(err).NotTo(HaveOccurred())
+				})
+			})
+
+			When("the pods have DNS names assigned and the coordinator have DNS names but DNS is disabled", func() {
+				BeforeEach(func() {
+					status.Client.Coordinators.Coordinators = nil
+
+					for _, process := range status.Cluster.Processes {
+						process.Locality[fdbv1beta2.FDBLocalityDNSNameKey] = process.Locality[fdbv1beta2.FDBLocalityZoneIDKey]
+						status.Client.Coordinators.Coordinators = append(status.Client.Coordinators.Coordinators,
+							fdbv1beta2.FoundationDBStatusCoordinator{
+								Address: fdbv1beta2.ProcessAddress{
+									StringAddress: process.Locality[fdbv1beta2.FDBLocalityZoneIDKey],
+									Port:          4501,
+								},
+							})
+					}
+
+					cluster.Spec.Routing.UseDNSInClusterFile = pointer.Bool(false)
+				})
+
+				It("should return that the coordinators are invalid", func() {
+					coordinatorsValid, addressesValid, err := CheckCoordinatorValidity(logr.Discard(), cluster, status, coordinatorStatus)
+					Expect(coordinatorsValid).To(BeFalse())
+					Expect(addressesValid).To(BeTrue())
+					Expect(err).NotTo(HaveOccurred())
+				})
+			})
+		})
+	})
+})
diff --git a/internal/locality/suite_test.go b/internal/locality/suite_test.go
new file mode 100644
index 00000000..127b750e
--- /dev/null
+++ b/internal/locality/suite_test.go
@@ -0,0 +1,33 @@
+/*
+ * suite_test.go
+ *
+ * This source file is part of the FoundationDB open source project
+ *
+ * Copyright 2022 Apple Inc. and the FoundationDB project authors
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package locality
+
+import (
+	"testing"
+
+	. "github.com/onsi/ginkgo/v2"
+	. "github.com/onsi/gomega"
+)
+
+func TestCmd(t *testing.T) {
+	RegisterFailHandler(Fail)
+	RunSpecs(t, "FDB locality")
+}
diff --git a/internal/test_helper.go b/internal/test_helper.go
index 63ae73f9..060dc308 100644
--- a/internal/test_helper.go
+++ b/internal/test_helper.go
@@ -29,9 +29,6 @@ import (
 
 // CreateDefaultCluster creates a default FoundationDBCluster for testing
 func CreateDefaultCluster() *fdbv1beta2.FoundationDBCluster {
-	trueValue := true
-	failureDetectionWindow := 1
-
 	return &fdbv1beta2.FoundationDBCluster{
 		ObjectMeta: metav1.ObjectMeta{
 			Name:      "operator-test-1",
@@ -48,8 +45,8 @@ func CreateDefaultCluster() *fdbv1beta2.FoundationDBCluster {
 			},
 			AutomationOptions: fdbv1beta2.FoundationDBClusterAutomationOptions{
 				Replacements: fdbv1beta2.AutomaticReplacementOptions{
-					Enabled:                     &trueValue,
-					FailureDetectionTimeSeconds: &failureDetectionWindow,
+					Enabled:                     pointer.Bool(true),
+					FailureDetectionTimeSeconds: pointer.Int(1),
 				},
 				WaitBetweenRemovalsSeconds: pointer.Int(0),
 			},