Repository: FoundationDB/fdb-kubernetes-operator
Pull request #1253 (2022-06-21 09:00:26)

Add changes to publish helm-chart
# Description

Successor of https://github.com/FoundationDB/fdb-kubernetes-operator/pull/899

## Type of change

*Please select one of the options below.*

- New feature (non-breaking change which adds functionality)

## Discussion

GH pages will be enabled this week.

## Testing

```bash
make chart-lint
docker run --rm -it -w /repo -v `pwd`:/repo quay.io/helmpack/chart-testing:v3.3.1 ct lint --all
Linting charts...
Version increment checking disabled.

------------------------------------------------------------------------------------------------------------------------
 Charts to be processed:
------------------------------------------------------------------------------------------------------------------------
 fdb-operator => (version: "0.2.0", path: "charts/fdb-operator")
------------------------------------------------------------------------------------------------------------------------

walk.go:74: found symbolic link in path: /repo/charts/fdb-operator/crds/apps.foundationdb.org_foundationdbbackups.yaml resolves to /repo/config/crd/bases/apps.foundationdb.org_foundationdbbackups.yaml
walk.go:74: found symbolic link in path: /repo/charts/fdb-operator/crds/apps.foundationdb.org_foundationdbclusters.yaml resolves to /repo/config/crd/bases/apps.foundationdb.org_foundationdbclusters.yaml
walk.go:74: found symbolic link in path: /repo/charts/fdb-operator/crds/apps.foundationdb.org_foundationdbrestores.yaml resolves to /repo/config/crd/bases/apps.foundationdb.org_foundationdbrestores.yaml
walk.go:74: found symbolic link in path: /repo/charts/fdb-operator/crds/apps.foundationdb.org_foundationdbbackups.yaml resolves to /repo/config/crd/bases/apps.foundationdb.org_foundationdbbackups.yaml
walk.go:74: found symbolic link in path: /repo/charts/fdb-operator/crds/apps.foundationdb.org_foundationdbclusters.yaml resolves to /repo/config/crd/bases/apps.foundationdb.org_foundationdbclusters.yaml
walk.go:74: found symbolic link in path: /repo/charts/fdb-operator/crds/apps.foundationdb.org_foundationdbrestores.yaml resolves to /repo/config/crd/bases/apps.foundationdb.org_foundationdbrestores.yaml
Linting chart 'fdb-operator => (version: "0.2.0", path: "charts/fdb-operator")'
Validating /repo/charts/fdb-operator/Chart.yaml...
Validation success! 👍
==> Linting charts/fdb-operator
[INFO] Chart.yaml: icon is recommended

1 chart(s) linted, 0 chart(s) failed
walk.go:74: found symbolic link in path: /repo/charts/fdb-operator/crds/apps.foundationdb.org_foundationdbbackups.yaml resolves to /repo/config/crd/bases/apps.foundationdb.org_foundationdbbackups.yaml
walk.go:74: found symbolic link in path: /repo/charts/fdb-operator/crds/apps.foundationdb.org_foundationdbclusters.yaml resolves to /repo/config/crd/bases/apps.foundationdb.org_foundationdbclusters.yaml
walk.go:74: found symbolic link in path: /repo/charts/fdb-operator/crds/apps.foundationdb.org_foundationdbrestores.yaml resolves to /repo/config/crd/bases/apps.foundationdb.org_foundationdbrestores.yaml
walk.go:74: found symbolic link in path: /repo/charts/fdb-operator/crds/apps.foundationdb.org_foundationdbbackups.yaml resolves to /repo/config/crd/bases/apps.foundationdb.org_foundationdbbackups.yaml
walk.go:74: found symbolic link in path: /repo/charts/fdb-operator/crds/apps.foundationdb.org_foundationdbclusters.yaml resolves to /repo/config/crd/bases/apps.foundationdb.org_foundationdbclusters.yaml
walk.go:74: found symbolic link in path: /repo/charts/fdb-operator/crds/apps.foundationdb.org_foundationdbrestores.yaml resolves to /repo/config/crd/bases/apps.foundationdb.org_foundationdbrestores.yaml
------------------------------------------------------------------------------------------------------------------------
 ✔︎ fdb-operator => (version: "0.2.0", path: "charts/fdb-operator")
------------------------------------------------------------------------------------------------------------------------
All charts linted successfully
```

## Documentation

Added some docs for the installation with helm.

## Follow-up

-

funkypenguin: Ooohyeah!
foundationdb-ci: 
### Result of fdb-kubernetes-operator-pr on Linux CentOS 7

* Commit ID: e0df1c32b56a92e646a4a6cf1835c10da2bedfab
* Duration 0:45:46
* Result: :x: **FAILED**
* Error: `Error while executing command: make -C tests -kj prOperator. Reason: exit status 2`
* [Build Logs](https://d1e0l78xbkh3xa.cloudfront.net/build-logs/fdb-kubernetes-operator/fdb-kubernetes-operator-pr/42743cf6-bef0-4bd1-b8ce-1ad35463e46c.gz) (available for 30 days)


Patch:
diff --git a/.github/workflows/on-push-lint-charts.yml b/.github/workflows/on-push-lint-charts.yml
new file mode 100644
index 00000000..11f18a92
--- /dev/null
+++ b/.github/workflows/on-push-lint-charts.yml
@@ -0,0 +1,57 @@
+name: Lint and Test Charts
+on:
+  push:
+    paths:
+      - 'charts/**'
+      - '.github/**'
+env:
+  HELM_VERSION: 3.5.2
+  CHECK_VERSION_INCREMENT: false # avoid tests failing when multiple attempts to push chart changes are made
+
+jobs:
+  lint-test:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v2
+        with:
+          fetch-depth: 0
+
+      - name: Set up Helm
+        uses: azure/setup-helm@v1
+        with:
+          version: v${{ env.HELM_VERSION }}
+
+      - name: Set up kube-score
+        run: |
+          wget https://github.com/zegl/kube-score/releases/download/v${{ env.KUBE_SCORE_VERSION }}/kube-score_${{ env.KUBE_SCORE_VERSION }}_linux_amd64 -O kube-score
+          chmod 755 kube-score
+
+      - name: Kube-score generated manifests
+        run: charts/.ci/scripts/local-kube-score.sh
+
+      # Python is a requirement for the chart-testing action below (supports yamllint among other tests)
+      - uses: actions/setup-python@v2
+        with:
+          python-version: 3.7
+
+      - name: Set up chart-testing
+        uses: helm/chart-testing-action@v2.2.1
+
+      - name: Run chart-testing (list-changed)
+        id: list-changed
+        run: |
+          changed=$(ct list-changed)
+          if [[ -n "$(ct list-changed)" ]]; then
+            echo "::set-output name=changed::true"
+          fi
+
+      - name: Run chart-testing (lint)
+        run: ct lint --check-version-increment=${{ env.CHECK_VERSION_INCREMENT }}
+
+      - name: Create kind cluster
+        uses: helm/kind-action@v1.3.0
+        if: steps.list-changed.outputs.changed == 'true'
+
+      - name: Run chart-testing (install)
+        run: ct install
diff --git a/.github/workflows/on-push-main-publish-chart.yml b/.github/workflows/on-push-main-publish-chart.yml
new file mode 100644
index 00000000..b8e5a9cb
--- /dev/null
+++ b/.github/workflows/on-push-main-publish-chart.yml
@@ -0,0 +1,36 @@
+name: Publish helm chart
+on:
+  push:
+    branches:
+      - main
+    paths:
+      - 'charts/**'
+      - '.github/**'
+env:
+  HELM_VERSION: 3.5.2
+
+jobs:
+  publish-chart:
+
+    runs-on: ubuntu-latest
+
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v2
+        with:
+          fetch-depth: 0
+
+      - name: Configure Git
+        run: |
+          git config user.name "$GITHUB_ACTOR"
+          git config user.email "$GITHUB_ACTOR@users.noreply.github.com"
+
+      - name: Install Helm
+        uses: azure/setup-helm@v1
+        with:
+          version: v${{ env.HELM_VERSION }}
+
+      - name: Run chart-releaser
+        uses: helm/chart-releaser-action@v1.2.0
+        env:
+          CR_TOKEN: "${{ secrets.GITHUB_TOKEN }}"
diff --git a/Makefile b/Makefile
index 554bc6fd..9eeb4926 100644
--- a/Makefile
+++ b/Makefile
@@ -2,7 +2,7 @@
 
 # Image URL to use all building/pushing image targets
 IMG ?= fdb-kubernetes-operator:latest
-SIDECAR_IMG ?= ""
+SIDECAR_IMG ?=
 REMOTE_BUILD ?= 0
 CRD_OPTIONS ?= "crd:maxDescLen=0,crdVersions=v1,generateEmbeddedObjectMeta=true"
 
@@ -245,3 +245,6 @@ bin/release: ${GO_SRC} $(GORELEASER)
 	$(GORELEASER) release --rm-dist
 	@mkdir -p bin
 	@touch $@
+
+chart-lint:
+	docker run --rm -it -w /repo -v `pwd`:/repo quay.io/helmpack/chart-testing:v3.3.1 ct lint --all
diff --git a/README.md b/README.md
index 0710876c..8326a768 100644
--- a/README.md
+++ b/README.md
@@ -38,6 +38,16 @@ For more information on version compatibility, see our [compatibility guide](doc
 
 For more information on the fields you can define on the cluster resource, see the [API documentation](docs/cluster_spec.md).
 
+### Using helm
+
+You can also use helm to install and manage the operator:
+
+```bash
+helm repo add fdb-kubernetes-operator https://foundationdb.github.io/fdb-kubernetes-operator/
+helm repo update
+helm install fdb-kubernetes-operator fdb-kubernetes-operator/fdb-kubernetes-operator
+ ```
+
 ## Local Development
 
 ### Environment Set-up
diff --git a/api/v1beta2/foundationdb_process_class.go b/api/v1beta2/foundationdb_process_class.go
index 16ac8952..d17bafa9 100644
--- a/api/v1beta2/foundationdb_process_class.go
+++ b/api/v1beta2/foundationdb_process_class.go
@@ -38,11 +38,13 @@ const (
 	ProcessClassClusterController ProcessClass = "cluster_controller"
 	// ProcessClassTest model for FDB class test
 	ProcessClassTest ProcessClass = "test"
+	// ProcessClassCoordinator model for FDB class coordinator
+	ProcessClassCoordinator ProcessClass = "coordinator"
 )
 
 // IsStateful determines whether a process class should store data.
 func (pClass ProcessClass) IsStateful() bool {
-	return pClass == ProcessClassStorage || pClass == ProcessClassLog || pClass == ProcessClassTransaction
+	return pClass == ProcessClassStorage || pClass == ProcessClassLog || pClass == ProcessClassTransaction || pClass == ProcessClassCoordinator
 }
 
 // IsTransaction determines whether a process class is part of the transaction system.
diff --git a/api/v1beta2/foundationdbcluster_types.go b/api/v1beta2/foundationdbcluster_types.go
index e1fb5013..8d71567d 100644
--- a/api/v1beta2/foundationdbcluster_types.go
+++ b/api/v1beta2/foundationdbcluster_types.go
@@ -1870,17 +1870,6 @@ func (cluster *FoundationDBCluster) NeedsReplacement(processGroup *ProcessGroupS
 		return false
 	}
 
-	// See: https://github.com/FoundationDB/fdb-kubernetes-operator/issues/1201 when a process group is set into crash loop
-	// we don't want to replace it (otherwise the newly created process group won't have that setting).
-	crashLoop, crashLoopAll := cluster.GetCrashLoopProcessGroups()
-	if crashLoopAll {
-		return false
-	}
-
-	if _, ok := crashLoop[processGroup.ProcessGroupID]; ok {
-		return false
-	}
-
 	if cluster.Spec.AutomationOptions.PodUpdateStrategy == PodUpdateStrategyReplacement {
 		return true
 	}
@@ -2035,3 +2024,32 @@ func (cluster *FoundationDBCluster) GetCrashLoopProcessGroups() (map[string]None
 
 	return crashLoopPods, false
 }
+
+// Validate checks if all settings in the cluster are valid, if not and error will be returned. If multiple issues are
+// found all of them will be returned in a single error.
+func (cluster *FoundationDBCluster) Validate() error {
+	var validations []string
+
+	// Check if the provided storage engine is valid for the defined FDB version.
+	version, err := ParseFdbVersion(cluster.Spec.Version)
+	if err != nil {
+		return err
+	}
+
+	if !version.IsStorageEngineSupported(cluster.Spec.DatabaseConfiguration.StorageEngine) {
+		validations = append(validations, fmt.Sprintf("storage engine %s is not supported on version %s", cluster.Spec.DatabaseConfiguration.StorageEngine, cluster.Spec.Version))
+	}
+
+	// Check if all coordinator processes are stateful
+	for _, selection := range cluster.Spec.CoordinatorSelection {
+		if !selection.ProcessClass.IsStateful() {
+			validations = append(validations, fmt.Sprintf("%s is not a valid process class for coordinators", selection.ProcessClass))
+		}
+	}
+
+	if len(validations) == 0 {
+		return nil
+	}
+
+	return fmt.Errorf(strings.Join(validations, ", "))
+}
diff --git a/helm/fdb-operator/Chart.yaml b/charts/fdb-operator/Chart.yaml
similarity index 98%
rename from helm/fdb-operator/Chart.yaml
rename to charts/fdb-operator/Chart.yaml
index 873f6ae2..80718326 100644
--- a/helm/fdb-operator/Chart.yaml
+++ b/charts/fdb-operator/Chart.yaml
@@ -21,4 +21,4 @@ version: 0.2.0
 
 # This is the version number of the application being deployed. This version number should be
 # incremented each time you make changes to the application.
-appVersion: 0.44.0
+appVersion: v1.6.0
diff --git a/helm/fdb-operator/crds/apps.foundationdb.org_foundationdbbackups.yaml b/charts/fdb-operator/crds/apps.foundationdb.org_foundationdbbackups.yaml
similarity index 100%
rename from helm/fdb-operator/crds/apps.foundationdb.org_foundationdbbackups.yaml
rename to charts/fdb-operator/crds/apps.foundationdb.org_foundationdbbackups.yaml
diff --git a/helm/fdb-operator/crds/apps.foundationdb.org_foundationdbclusters.yaml b/charts/fdb-operator/crds/apps.foundationdb.org_foundationdbclusters.yaml
similarity index 100%
rename from helm/fdb-operator/crds/apps.foundationdb.org_foundationdbclusters.yaml
rename to charts/fdb-operator/crds/apps.foundationdb.org_foundationdbclusters.yaml
diff --git a/helm/fdb-operator/crds/apps.foundationdb.org_foundationdbrestores.yaml b/charts/fdb-operator/crds/apps.foundationdb.org_foundationdbrestores.yaml
similarity index 100%
rename from helm/fdb-operator/crds/apps.foundationdb.org_foundationdbrestores.yaml
rename to charts/fdb-operator/crds/apps.foundationdb.org_foundationdbrestores.yaml
diff --git a/helm/fdb-operator/templates/NOTES.txt b/charts/fdb-operator/templates/NOTES.txt
similarity index 100%
rename from helm/fdb-operator/templates/NOTES.txt
rename to charts/fdb-operator/templates/NOTES.txt
diff --git a/helm/fdb-operator/templates/_helpers.tpl b/charts/fdb-operator/templates/_helpers.tpl
similarity index 100%
rename from helm/fdb-operator/templates/_helpers.tpl
rename to charts/fdb-operator/templates/_helpers.tpl
diff --git a/helm/fdb-operator/templates/manager/deployment.yaml b/charts/fdb-operator/templates/manager/deployment.yaml
similarity index 100%
rename from helm/fdb-operator/templates/manager/deployment.yaml
rename to charts/fdb-operator/templates/manager/deployment.yaml
diff --git a/helm/fdb-operator/templates/rbac/rbac_role.yaml b/charts/fdb-operator/templates/rbac/rbac_role.yaml
similarity index 100%
rename from helm/fdb-operator/templates/rbac/rbac_role.yaml
rename to charts/fdb-operator/templates/rbac/rbac_role.yaml
diff --git a/helm/fdb-operator/templates/rbac/rbac_role_binding.yaml b/charts/fdb-operator/templates/rbac/rbac_role_binding.yaml
similarity index 100%
rename from helm/fdb-operator/templates/rbac/rbac_role_binding.yaml
rename to charts/fdb-operator/templates/rbac/rbac_role_binding.yaml
diff --git a/helm/fdb-operator/templates/rbac/serviceaccount.yaml b/charts/fdb-operator/templates/rbac/serviceaccount.yaml
similarity index 100%
rename from helm/fdb-operator/templates/rbac/serviceaccount.yaml
rename to charts/fdb-operator/templates/rbac/serviceaccount.yaml
diff --git a/helm/fdb-operator/templates/tests/test-connection.yaml b/charts/fdb-operator/templates/tests/test-connection.yaml
similarity index 100%
rename from helm/fdb-operator/templates/tests/test-connection.yaml
rename to charts/fdb-operator/templates/tests/test-connection.yaml
diff --git a/helm/fdb-operator/values.yaml b/charts/fdb-operator/values.yaml
similarity index 100%
rename from helm/fdb-operator/values.yaml
rename to charts/fdb-operator/values.yaml
diff --git a/controllers/cluster_controller.go b/controllers/cluster_controller.go
index 4ce472bd..6fe79f6f 100644
--- a/controllers/cluster_controller.go
+++ b/controllers/cluster_controller.go
@@ -120,13 +120,10 @@ func (r *FoundationDBClusterReconciler) Reconcile(ctx context.Context, request c
 		return ctrl.Result{}, fmt.Errorf("version %s is not supported", cluster.Spec.Version)
 	}
 
-	storageEngineSupported, err := isStorageEngineSupported(cluster.Spec.Version, cluster.Spec.DatabaseConfiguration.StorageEngine)
+	err = cluster.Validate()
 	if err != nil {
-		return ctrl.Result{}, err
-	}
-	if !storageEngineSupported {
-		r.Recorder.Event(cluster, corev1.EventTypeWarning, "Storage engine not supported", fmt.Sprintf("storage engine %s is not supported on version %s", cluster.Spec.DatabaseConfiguration.StorageEngine, cluster.Spec.Version))
-		return ctrl.Result{}, fmt.Errorf("storage engine %s is not supported on version %s", cluster.Spec.DatabaseConfiguration.StorageEngine, cluster.Spec.Version)
+		r.Recorder.Event(cluster, corev1.EventTypeWarning, "ClusterSpec not valid", err.Error())
+		return ctrl.Result{}, fmt.Errorf("ClusterSpec is not valid: %w", err)
 	}
 
 	subReconcilers := []clusterSubReconciler{
@@ -134,9 +131,9 @@ func (r *FoundationDBClusterReconciler) Reconcile(ctx context.Context, request c
 		updateLockConfiguration{},
 		updateConfigMap{},
 		checkClientCompatibility{},
+		deletePodsForBuggification{},
 		replaceMisconfiguredProcessGroups{},
 		replaceFailedProcessGroups{},
-		deletePodsForBuggification{},
 		addProcessGroups{},
 		addServices{},
 		addPVCs{},
@@ -750,11 +747,3 @@ func (r *FoundationDBClusterReconciler) getCoordinatorSet(cluster *fdbv1beta2.Fo
 
 	return adminClient.GetCoordinatorSet()
 }
-
-func isStorageEngineSupported(versionString string, storageEngine fdbv1beta2.StorageEngine) (bool, error) {
-	version, err := fdbv1beta2.ParseFdbVersion(versionString)
-	if err != nil {
-		return false, err
-	}
-	return version.IsStorageEngineSupported(storageEngine), nil
-}
diff --git a/controllers/delete_pods_for_buggification.go b/controllers/delete_pods_for_buggification.go
index 26a4b1e5..9e960e21 100644
--- a/controllers/delete_pods_for_buggification.go
+++ b/controllers/delete_pods_for_buggification.go
@@ -44,10 +44,13 @@ func (d deletePodsForBuggification) reconcile(ctx context.Context, r *Foundation
 	}
 
 	podMap := internal.CreatePodMap(cluster, pods)
-	updates := make([]*corev1.Pod, 0)
-
 	crashLoopPods, crashLoopAll := cluster.GetCrashLoopProcessGroups()
+	noSchedulePods := make(map[string]fdbv1beta2.None, len(cluster.Spec.Buggify.NoSchedule))
+	for _, processGroupID := range cluster.Spec.Buggify.NoSchedule {
+		noSchedulePods[processGroupID] = fdbv1beta2.None{}
+	}
 
+	var updates []*corev1.Pod
 	for _, processGroup := range cluster.Status.ProcessGroups {
 		if processGroup.IsMarkedForRemoval() {
 			logger.V(1).Info("Ignore process group marked for removal",
@@ -80,6 +83,42 @@ func (d deletePodsForBuggification) reconcile(ctx context.Context, r *Foundation
 				"shouldCrashLoop", shouldCrashLoop,
 				"inCrashLoop", inCrashLoop)
 			updates = append(updates, pod)
+			continue
+		}
+
+		// Recreate Pods that should be in the no schedule state
+		var inNoSchedule, shouldBeNoSchedule bool
+		_, shouldBeNoSchedule = noSchedulePods[processGroup.ProcessGroupID]
+
+		// Ensure the Pod has an Affinity set before accessing it.
+		if pod.Spec.Affinity == nil ||
+			pod.Spec.Affinity.NodeAffinity == nil ||
+			pod.Spec.Affinity.NodeAffinity.RequiredDuringSchedulingIgnoredDuringExecution == nil {
+			// Uf the Pod should be in the noSchedule state we have to recreate the Pod.
+			if shouldBeNoSchedule {
+				logger.Info("Deleting pod for buggification",
+					"processGroupID", processGroup.ProcessGroupID,
+					"shouldBeNoSchedule", shouldBeNoSchedule,
+					"inNoSchedule", inNoSchedule)
+				updates = append(updates, pod)
+			}
+			continue
+		}
+
+		for _, term := range pod.Spec.Affinity.NodeAffinity.RequiredDuringSchedulingIgnoredDuringExecution.NodeSelectorTerms {
+			for _, expression := range term.MatchExpressions {
+				if expression.Key == fdbv1beta2.NodeSelectorNoScheduleLabel {
+					inNoSchedule = true
+				}
+			}
+		}
+
+		if inNoSchedule != shouldBeNoSchedule {
+			logger.Info("Deleting pod for buggification",
+				"processGroupID", processGroup.ProcessGroupID,
+				"shouldBeNoSchedule", shouldBeNoSchedule,
+				"inNoSchedule", inNoSchedule)
+			updates = append(updates, pod)
 		}
 	}
 
@@ -94,5 +133,6 @@ func (d deletePodsForBuggification) reconcile(ctx context.Context, r *Foundation
 
 		return &requeue{message: "Pods need to be recreated"}
 	}
+
 	return nil
 }
diff --git a/controllers/remove_process_groups.go b/controllers/remove_process_groups.go
index 579268e0..ca85ed6c 100644
--- a/controllers/remove_process_groups.go
+++ b/controllers/remove_process_groups.go
@@ -244,9 +244,7 @@ func includeProcessGroup(ctx context.Context, r *FoundationDBClusterReconciler,
 	}
 	defer adminClient.Close()
 
-	var fdbProcessesToInclude []fdbv1beta2.ProcessAddress
-	var processGroups []*fdbv1beta2.ProcessGroupStatus
-	fdbProcessesToInclude, processGroups = getProcessesToInclude(cluster, removedProcessGroups)
+	fdbProcessesToInclude := getProcessesToInclude(cluster, removedProcessGroups)
 	if len(fdbProcessesToInclude) > 0 {
 		r.Recorder.Event(cluster, corev1.EventTypeNormal, "IncludingProcesses", fmt.Sprintf("Including removed processes: %v", fdbProcessesToInclude))
 
@@ -255,7 +253,6 @@ func includeProcessGroup(ctx context.Context, r *FoundationDBClusterReconciler,
 			return err
 		}
 
-		cluster.Status.ProcessGroups = processGroups
 		err := r.Status().Update(ctx, cluster)
 		if err != nil {
 			return err
@@ -265,11 +262,14 @@ func includeProcessGroup(ctx context.Context, r *FoundationDBClusterReconciler,
 	return nil
 }
 
-func getProcessesToInclude(cluster *fdbv1beta2.FoundationDBCluster, removedProcessGroups map[string]bool) ([]fdbv1beta2.ProcessAddress, []*fdbv1beta2.ProcessGroupStatus) {
+func getProcessesToInclude(cluster *fdbv1beta2.FoundationDBCluster, removedProcessGroups map[string]bool) []fdbv1beta2.ProcessAddress {
 	fdbProcessesToInclude := make([]fdbv1beta2.ProcessAddress, 0)
-	// TODO (09harsh): don't allocate extra space for processGroups, #1196
-	processGroups := make([]*fdbv1beta2.ProcessGroupStatus, 0, len(cluster.Status.ProcessGroups))
 
+	if len(removedProcessGroups) == 0 {
+		return fdbProcessesToInclude
+	}
+
+	idx := 0
 	for _, processGroup := range cluster.Status.ProcessGroups {
 		if processGroup.IsMarkedForRemoval() && removedProcessGroups[processGroup.ProcessGroupID] {
 			if cluster.UseLocalitiesForExclusion() {
@@ -278,11 +278,16 @@ func getProcessesToInclude(cluster *fdbv1beta2.FoundationDBCluster, removedProce
 			for _, pAddr := range processGroup.Addresses {
 				fdbProcessesToInclude = append(fdbProcessesToInclude, fdbv1beta2.ProcessAddress{IPAddress: net.ParseIP(pAddr)})
 			}
-		} else {
-			processGroups = append(processGroups, processGroup)
+			continue
 		}
+		cluster.Status.ProcessGroups[idx] = processGroup
+		idx++
 	}
-	return fdbProcessesToInclude, processGroups
+
+	// Remove the trailing duplicates.
+	cluster.Status.ProcessGroups = cluster.Status.ProcessGroups[:idx]
+
+	return fdbProcessesToInclude
 }
 
 func (r *FoundationDBClusterReconciler) getProcessGroupsToRemove(cluster *fdbv1beta2.FoundationDBCluster, remainingMap map[string]bool) (bool, bool, []*fdbv1beta2.ProcessGroupStatus) {
diff --git a/ct.yaml b/ct.yaml
new file mode 100644
index 00000000..5ebc892e
--- /dev/null
+++ b/ct.yaml
@@ -0,0 +1,3 @@
+chart-dirs:
+    - charts
+validate-maintainers: false
diff --git a/docs/manual/fault_domains.md b/docs/manual/fault_domains.md
index 195dc024..80a668ef 100644
--- a/docs/manual/fault_domains.md
+++ b/docs/manual/fault_domains.md
@@ -246,6 +246,13 @@ In this example the processes with the class `storage` will be preferred over pr
 That means that a `log` process will only be considered a valid coordinator if there are no other `storage` processes that can be selected without hurting the fault domain requirements.
 Changing the `coordinatorSelection` can result in new coordinators e.g. if the current preferred class will be removed.
 
+The operator supports the following classes as coordinators:
+
+- `storage`
+- `log`
+- `transaction`
+- `coordinator`
+
 ### Known limitations
 
 FoundationDB clusters that are spread across different DC's or Kubernetes clusters only support the same `coordinatorSelection`.


Test patch:
diff --git a/api/v1beta2/foundationdbcluster_types_test.go b/api/v1beta2/foundationdbcluster_types_test.go
index 09811eed..92d299ef 100644
--- a/api/v1beta2/foundationdbcluster_types_test.go
+++ b/api/v1beta2/foundationdbcluster_types_test.go
@@ -4433,44 +4433,6 @@ var _ = Describe("[api] FoundationDBCluster", func() {
 				},
 				false,
 			),
-			Entry("Default update strategy log process that is in crash loop",
-				&FoundationDBCluster{
-					Spec: FoundationDBClusterSpec{
-						AutomationOptions: FoundationDBClusterAutomationOptions{
-							PodUpdateStrategy: "",
-						},
-						Buggify: BuggifyConfig{
-							CrashLoop: []string{
-								"log-1",
-							},
-						},
-					},
-				},
-				&ProcessGroupStatus{
-					ProcessGroupID: "log-1",
-					ProcessClass:   ProcessClassTransaction,
-				},
-				false,
-			),
-			Entry("Default update strategy log process with crash loop all",
-				&FoundationDBCluster{
-					Spec: FoundationDBClusterSpec{
-						AutomationOptions: FoundationDBClusterAutomationOptions{
-							PodUpdateStrategy: "",
-						},
-						Buggify: BuggifyConfig{
-							CrashLoop: []string{
-								"*",
-							},
-						},
-					},
-				},
-				&ProcessGroupStatus{
-					ProcessGroupID: "log-1",
-					ProcessClass:   ProcessClassTransaction,
-				},
-				false,
-			),
 		)
 	})
 
@@ -4551,4 +4513,119 @@ var _ = Describe("[api] FoundationDBCluster", func() {
 			})
 		})
 	})
+
+	When("validating a cluster", func() {
+		DescribeTable("it should return if the cluster is valid",
+			func(cluster *FoundationDBCluster, expected error) {
+				if expected == nil {
+					Expect(cluster.Validate()).NotTo(HaveOccurred())
+				} else {
+					Expect(cluster.Validate()).To(Equal(expected))
+				}
+
+			},
+			Entry("valid cluster spec",
+				&FoundationDBCluster{
+					Spec: FoundationDBClusterSpec{
+						Version: "6.3.2",
+						DatabaseConfiguration: DatabaseConfiguration{
+							StorageEngine: StorageEngineSSD2,
+						},
+					},
+				},
+				nil,
+			),
+			Entry("using invalid storage engine",
+				&FoundationDBCluster{
+					Spec: FoundationDBClusterSpec{
+						Version: "6.3.2",
+						DatabaseConfiguration: DatabaseConfiguration{
+							StorageEngine: StorageEngineRocksDbV1,
+						},
+					},
+				},
+				fmt.Errorf("storage engine ssd-rocksdb-v1 is not supported on version 6.3.2"),
+			),
+			Entry("using valid storage engine",
+				&FoundationDBCluster{
+					Spec: FoundationDBClusterSpec{
+						Version: Versions.SupportsRocksDBV1.String(),
+						DatabaseConfiguration: DatabaseConfiguration{
+							StorageEngine: StorageEngineRocksDbV1,
+						},
+					},
+				},
+				nil,
+			),
+			Entry("using valid coordinator selection",
+				&FoundationDBCluster{
+					Spec: FoundationDBClusterSpec{
+						Version: Versions.SupportsRocksDBV1.String(),
+						DatabaseConfiguration: DatabaseConfiguration{
+							StorageEngine: StorageEngineRocksDbV1,
+						},
+						CoordinatorSelection: []CoordinatorSelectionSetting{
+							{
+								ProcessClass: ProcessClassStorage,
+							},
+							{
+								ProcessClass: ProcessClassLog,
+							},
+							{
+								ProcessClass: ProcessClassCoordinator,
+							},
+							{
+								ProcessClass: ProcessClassTransaction,
+							},
+						},
+					},
+				},
+				nil,
+			),
+			Entry("using invalid coordinator selection",
+				&FoundationDBCluster{
+					Spec: FoundationDBClusterSpec{
+						Version: Versions.SupportsRocksDBV1.String(),
+						DatabaseConfiguration: DatabaseConfiguration{
+							StorageEngine: StorageEngineRocksDbV1,
+						},
+						CoordinatorSelection: []CoordinatorSelectionSetting{
+							{
+								ProcessClass: ProcessClassStorage,
+							},
+							{
+								ProcessClass: ProcessClassLog,
+							},
+							{
+								ProcessClass: ProcessClassCoordinator,
+							},
+							{
+								ProcessClass: ProcessClassTransaction,
+							},
+							{
+								ProcessClass: ProcessClassStateless,
+							},
+						},
+					},
+				},
+				fmt.Errorf("stateless is not a valid process class for coordinators"),
+			),
+			Entry("multiple validations",
+				&FoundationDBCluster{
+					Spec: FoundationDBClusterSpec{
+						Version: "6.1.3",
+						DatabaseConfiguration: DatabaseConfiguration{
+							StorageEngine: StorageEngineRocksDbV1,
+						},
+						CoordinatorSelection: []CoordinatorSelectionSetting{
+							{
+								ProcessClass: ProcessClassStateless,
+							},
+						},
+					},
+				},
+				fmt.Errorf("storage engine ssd-rocksdb-v1 is not supported on version 6.1.3, stateless is not a valid process class for coordinators"),
+			),
+		)
+	})
 })
diff --git a/controllers/delete_pods_for_buggification_test.go b/controllers/delete_pods_for_buggification_test.go
index 66168c3b..f51f2546 100644
--- a/controllers/delete_pods_for_buggification_test.go
+++ b/controllers/delete_pods_for_buggification_test.go
@@ -87,25 +87,50 @@ var _ = Describe("delete_pods_for_buggification", func() {
 	})
 
 	Context("with a buggification that needs to be enabled", func() {
-		BeforeEach(func() {
-			cluster.Spec.Buggify.CrashLoop = []string{"storage-1"}
-		})
-
-		It("should requeue", func() {
-			Expect(requeue).NotTo(BeNil())
-			Expect(requeue.message).To(Equal("Pods need to be recreated"))
+		When("using crashLoop", func() {
+			BeforeEach(func() {
+				cluster.Spec.Buggify.CrashLoop = []string{"storage-1"}
+			})
+
+			It("should requeue", func() {
+				Expect(requeue).NotTo(BeNil())
+				Expect(requeue.message).To(Equal("Pods need to be recreated"))
+			})
+
+			It("should delete the pod", func() {
+				pods := &corev1.PodList{}
+				err = k8sClient.List(context.TODO(), pods, getListOptions(cluster)...)
+				Expect(err).NotTo(HaveOccurred())
+				Expect(len(pods.Items)).To(Equal(len(originalPods.Items) - 1))
+
+				pod := &corev1.Pod{}
+				err = k8sClient.Get(context.TODO(), types.NamespacedName{Namespace: cluster.Namespace, Name: "operator-test-1-storage-1"}, pod)
+				Expect(err).To(HaveOccurred())
+				Expect(k8serrors.IsNotFound(err)).To(BeTrue())
+			})
 		})
 
-		It("should delete the pod", func() {
-			pods := &corev1.PodList{}
-			err = k8sClient.List(context.TODO(), pods, getListOptions(cluster)...)
-			Expect(err).NotTo(HaveOccurred())
-			Expect(len(pods.Items)).To(Equal(len(originalPods.Items) - 1))
-
-			pod := &corev1.Pod{}
-			err = k8sClient.Get(context.TODO(), types.NamespacedName{Namespace: cluster.Namespace, Name: "operator-test-1-storage-1"}, pod)
-			Expect(err).To(HaveOccurred())
-			Expect(k8serrors.IsNotFound(err)).To(BeTrue())
+		When("using noSchedule", func() {
+			BeforeEach(func() {
+				cluster.Spec.Buggify.NoSchedule = []string{"storage-1"}
+			})
+
+			It("should requeue", func() {
+				Expect(requeue).NotTo(BeNil())
+				Expect(requeue.message).To(Equal("Pods need to be recreated"))
+			})
+
+			It("should delete the pod", func() {
+				pods := &corev1.PodList{}
+				err = k8sClient.List(context.TODO(), pods, getListOptions(cluster)...)
+				Expect(err).NotTo(HaveOccurred())
+				Expect(len(pods.Items)).To(Equal(len(originalPods.Items) - 1))
+
+				pod := &corev1.Pod{}
+				err = k8sClient.Get(context.TODO(), types.NamespacedName{Namespace: cluster.Namespace, Name: "operator-test-1-storage-1"}, pod)
+				Expect(err).To(HaveOccurred())
+				Expect(k8serrors.IsNotFound(err)).To(BeTrue())
+			})
 		})
 	})
 
@@ -133,65 +158,162 @@ var _ = Describe("delete_pods_for_buggification", func() {
 	})
 
 	Context("with a buggification that needs to be disabled", func() {
-		BeforeEach(func() {
-			pod := &corev1.Pod{}
-			err = k8sClient.Get(context.TODO(), types.NamespacedName{Namespace: cluster.Namespace, Name: "operator-test-1-storage-1"}, pod)
-			Expect(err).NotTo(HaveOccurred())
-
-			err = k8sClient.Delete(context.TODO(), pod)
-			Expect(err).NotTo(HaveOccurred())
-
-			pod.Spec.Containers[0].Args = []string{"crash-loop"}
-			err = k8sClient.Create(context.TODO(), pod)
-			Expect(err).NotTo(HaveOccurred())
+		When("using crashLoop", func() {
+			BeforeEach(func() {
+				pod := &corev1.Pod{}
+				err = k8sClient.Get(context.TODO(), types.NamespacedName{Namespace: cluster.Namespace, Name: "operator-test-1-storage-1"}, pod)
+				Expect(err).NotTo(HaveOccurred())
+
+				err = k8sClient.Delete(context.TODO(), pod)
+				Expect(err).NotTo(HaveOccurred())
+
+				pod.Spec.Containers[0].Args = []string{"crash-loop"}
+				err = k8sClient.Create(context.TODO(), pod)
+				Expect(err).NotTo(HaveOccurred())
+			})
+
+			It("should requeue", func() {
+				Expect(requeue).NotTo(BeNil())
+				Expect(requeue.message).To(Equal("Pods need to be recreated"))
+			})
+
+			It("should delete the pod", func() {
+				pods := &corev1.PodList{}
+				err = k8sClient.List(context.TODO(), pods, getListOptions(cluster)...)
+				Expect(err).NotTo(HaveOccurred())
+				Expect(len(pods.Items)).To(Equal(len(originalPods.Items) - 1))
+
+				pod := &corev1.Pod{}
+				err = k8sClient.Get(context.TODO(), types.NamespacedName{Namespace: cluster.Namespace, Name: "operator-test-1-storage-1"}, pod)
+				Expect(err).To(HaveOccurred())
+				Expect(k8serrors.IsNotFound(err)).To(BeTrue())
+			})
 		})
 
-		It("should requeue", func() {
-			Expect(requeue).NotTo(BeNil())
-			Expect(requeue.message).To(Equal("Pods need to be recreated"))
-		})
+		When("using noSchedule", func() {
+			BeforeEach(func() {
+				pod := &corev1.Pod{}
+				err = k8sClient.Get(context.TODO(), types.NamespacedName{Namespace: cluster.Namespace, Name: "operator-test-1-storage-1"}, pod)
+				Expect(err).NotTo(HaveOccurred())
 
-		It("should delete the pod", func() {
-			pods := &corev1.PodList{}
-			err = k8sClient.List(context.TODO(), pods, getListOptions(cluster)...)
-			Expect(err).NotTo(HaveOccurred())
-			Expect(len(pods.Items)).To(Equal(len(originalPods.Items) - 1))
+				err = k8sClient.Delete(context.TODO(), pod)
+				Expect(err).NotTo(HaveOccurred())
 
-			pod := &corev1.Pod{}
-			err = k8sClient.Get(context.TODO(), types.NamespacedName{Namespace: cluster.Namespace, Name: "operator-test-1-storage-1"}, pod)
-			Expect(err).To(HaveOccurred())
-			Expect(k8serrors.IsNotFound(err)).To(BeTrue())
+				pod.Spec.Affinity = &corev1.Affinity{
+					NodeAffinity: &corev1.NodeAffinity{
+						RequiredDuringSchedulingIgnoredDuringExecution: &corev1.NodeSelector{
+							NodeSelectorTerms: []corev1.NodeSelectorTerm{
+								{
+									MatchExpressions: []corev1.NodeSelectorRequirement{
+										{
+											Key: fdbv1beta2.NodeSelectorNoScheduleLabel,
+										},
+									},
+								},
+							},
+						},
+					},
+				}
+
+				err = k8sClient.Create(context.TODO(), pod)
+				Expect(err).NotTo(HaveOccurred())
+			})
+
+			It("should requeue", func() {
+				Expect(requeue).NotTo(BeNil())
+				Expect(requeue.message).To(Equal("Pods need to be recreated"))
+			})
+
+			It("should delete the pod", func() {
+				pods := &corev1.PodList{}
+				err = k8sClient.List(context.TODO(), pods, getListOptions(cluster)...)
+				Expect(err).NotTo(HaveOccurred())
+				Expect(len(pods.Items)).To(Equal(len(originalPods.Items) - 1))
+
+				pod := &corev1.Pod{}
+				err = k8sClient.Get(context.TODO(), types.NamespacedName{Namespace: cluster.Namespace, Name: "operator-test-1-storage-1"}, pod)
+				Expect(err).To(HaveOccurred())
+				Expect(k8serrors.IsNotFound(err)).To(BeTrue())
+			})
 		})
 	})
 
 	Context("with a buggification that is already on", func() {
-		BeforeEach(func() {
-			pod := &corev1.Pod{}
-			err = k8sClient.Get(context.TODO(), types.NamespacedName{Namespace: cluster.Namespace, Name: "operator-test-1-storage-1"}, pod)
-			Expect(err).NotTo(HaveOccurred())
-
-			err = k8sClient.Delete(context.TODO(), pod)
-			Expect(err).NotTo(HaveOccurred())
-
-			pod.Spec.Containers[0].Args = []string{"crash-loop"}
-			err = k8sClient.Create(context.TODO(), pod)
-			Expect(err).NotTo(HaveOccurred())
-			cluster.Spec.Buggify.CrashLoop = []string{"storage-1"}
+		When("using crashLoop", func() {
+			BeforeEach(func() {
+				pod := &corev1.Pod{}
+				err = k8sClient.Get(context.TODO(), types.NamespacedName{Namespace: cluster.Namespace, Name: "operator-test-1-storage-1"}, pod)
+				Expect(err).NotTo(HaveOccurred())
+
+				err = k8sClient.Delete(context.TODO(), pod)
+				Expect(err).NotTo(HaveOccurred())
+
+				pod.Spec.Containers[0].Args = []string{"crash-loop"}
+				err = k8sClient.Create(context.TODO(), pod)
+				Expect(err).NotTo(HaveOccurred())
+				cluster.Spec.Buggify.CrashLoop = []string{"storage-1"}
+			})
+
+			It("should not requeue", func() {
+				Expect(requeue).To(BeNil())
+			})
+
+			It("should not delete any pods", func() {
+				pods := &corev1.PodList{}
+				err = k8sClient.List(context.TODO(), pods, getListOptions(cluster)...)
+				Expect(err).NotTo(HaveOccurred())
+				Expect(len(pods.Items)).To(Equal(len(originalPods.Items)))
+
+				pod := &corev1.Pod{}
+				err = k8sClient.Get(context.TODO(), types.NamespacedName{Namespace: cluster.Namespace, Name: "operator-test-1-storage-1"}, pod)
+				Expect(err).NotTo(HaveOccurred())
+			})
 		})
 
-		It("should not requeue", func() {
-			Expect(requeue).To(BeNil())
-		})
+		When("using noSchedule", func() {
+			BeforeEach(func() {
+				pod := &corev1.Pod{}
+				err = k8sClient.Get(context.TODO(), types.NamespacedName{Namespace: cluster.Namespace, Name: "operator-test-1-storage-1"}, pod)
+				Expect(err).NotTo(HaveOccurred())
 
-		It("should not delete any pods", func() {
-			pods := &corev1.PodList{}
-			err = k8sClient.List(context.TODO(), pods, getListOptions(cluster)...)
-			Expect(err).NotTo(HaveOccurred())
-			Expect(len(pods.Items)).To(Equal(len(originalPods.Items)))
+				err = k8sClient.Delete(context.TODO(), pod)
+				Expect(err).NotTo(HaveOccurred())
 
-			pod := &corev1.Pod{}
-			err = k8sClient.Get(context.TODO(), types.NamespacedName{Namespace: cluster.Namespace, Name: "operator-test-1-storage-1"}, pod)
-			Expect(err).NotTo(HaveOccurred())
+				pod.Spec.Affinity = &corev1.Affinity{
+					NodeAffinity: &corev1.NodeAffinity{
+						RequiredDuringSchedulingIgnoredDuringExecution: &corev1.NodeSelector{
+							NodeSelectorTerms: []corev1.NodeSelectorTerm{
+								{
+									MatchExpressions: []corev1.NodeSelectorRequirement{
+										{
+											Key: fdbv1beta2.NodeSelectorNoScheduleLabel,
+										},
+									},
+								},
+							},
+						},
+					},
+				}
+
+				err = k8sClient.Create(context.TODO(), pod)
+				Expect(err).NotTo(HaveOccurred())
+				cluster.Spec.Buggify.NoSchedule = []string{"storage-1"}
+			})
+
+			It("should not requeue", func() {
+				Expect(requeue).To(BeNil())
+			})
+
+			It("should not delete any pods", func() {
+				pods := &corev1.PodList{}
+				err = k8sClient.List(context.TODO(), pods, getListOptions(cluster)...)
+				Expect(err).NotTo(HaveOccurred())
+				Expect(len(pods.Items)).To(Equal(len(originalPods.Items)))
+
+				pod := &corev1.Pod{}
+				err = k8sClient.Get(context.TODO(), types.NamespacedName{Namespace: cluster.Namespace, Name: "operator-test-1-storage-1"}, pod)
+				Expect(err).NotTo(HaveOccurred())
+			})
 		})
 	})
 
diff --git a/controllers/remove_process_groups_test.go b/controllers/remove_process_groups_test.go
index 1b5cfee5..543cccfc 100644
--- a/controllers/remove_process_groups_test.go
+++ b/controllers/remove_process_groups_test.go
@@ -328,9 +328,7 @@ var _ = Describe("remove_process_groups", func() {
 
 			When("including no process", func() {
 				It("should not include any process", func() {
-					fdbProcessesToInclude, processGroups := getProcessesToInclude(cluster, removedProcessGroups)
-					Expect(len(fdbProcessesToInclude)).To(Equal(0))
-					Expect(len(processGroups)).To(Equal(16))
+					Expect(len(getProcessesToInclude(cluster, removedProcessGroups))).To(Equal(0))
 				})
 			})
 
@@ -345,10 +343,10 @@ var _ = Describe("remove_process_groups", func() {
 				})
 
 				It("should include one process", func() {
-					fdbProcessesToInclude, processGroups := getProcessesToInclude(cluster, removedProcessGroups)
+					fdbProcessesToInclude := getProcessesToInclude(cluster, removedProcessGroups)
 					Expect(len(fdbProcessesToInclude)).To(Equal(1))
 					Expect(fdbv1beta2.ProcessAddressesString(fdbProcessesToInclude, " ")).To(Equal("1.1.1.1"))
-					Expect(len(processGroups)).To(Equal(15))
+					Expect(len(cluster.Status.ProcessGroups)).To(Equal(15))
 				})
 			})
 		})
@@ -360,27 +358,29 @@ var _ = Describe("remove_process_groups", func() {
 
 			When("including no process", func() {
 				It("should not include any process", func() {
-					fdbProcessesToInclude, processGroups := getProcessesToInclude(cluster, removedProcessGroups)
+					fdbProcessesToInclude := getProcessesToInclude(cluster, removedProcessGroups)
 					Expect(len(fdbProcessesToInclude)).To(Equal(0))
-					Expect(len(processGroups)).To(Equal(16))
+					Expect(len(cluster.Status.ProcessGroups)).To(Equal(16))
 				})
 			})
 
 			When("including one process", func() {
+				var removedProcessGroup *fdbv1beta2.ProcessGroupStatus
+
 				BeforeEach(func() {
-					processGroup := cluster.Status.ProcessGroups[0]
-					Expect(processGroup.ProcessGroupID).To(Equal("storage-1"))
-					processGroup.MarkForRemoval()
-					cluster.Status.ProcessGroups[0] = processGroup
+					removedProcessGroup = cluster.Status.ProcessGroups[0]
+					Expect(removedProcessGroup.ProcessGroupID).To(Equal("storage-1"))
+					removedProcessGroup.MarkForRemoval()
+					cluster.Status.ProcessGroups[0] = removedProcessGroup
 
-					removedProcessGroups[processGroup.ProcessGroupID] = true
+					removedProcessGroups[removedProcessGroup.ProcessGroupID] = true
 				})
 
 				It("should include one process", func() {
-					fdbProcessesToInclude, processGroups := getProcessesToInclude(cluster, removedProcessGroups)
+					fdbProcessesToInclude := getProcessesToInclude(cluster, removedProcessGroups)
 					Expect(len(fdbProcessesToInclude)).To(Equal(2))
-					Expect(fdbv1beta2.ProcessAddressesString(fdbProcessesToInclude, " ")).To(Equal(fmt.Sprintf("%s %s", cluster.Status.ProcessGroups[0].GetExclusionString(), cluster.Status.ProcessGroups[0].Addresses[0])))
-					Expect(len(processGroups)).To(Equal(15))
+					Expect(fdbv1beta2.ProcessAddressesString(fdbProcessesToInclude, " ")).To(Equal(fmt.Sprintf("%s %s", removedProcessGroup.GetExclusionString(), removedProcessGroup.Addresses[0])))
+					Expect(len(cluster.Status.ProcessGroups)).To(Equal(15))
 				})
 			})
 		})