Repository: k8ssandra/k8ssandra-operator (go)

Issue:
K8SSAND-1170 ⁃ e2e test flakes in Stargate tests
<!--

Feel free to ask questions on Discord at https://discord.com/invite/y4s64xC9 or in the forum at https://forum.k8ssandra.io/. 
-->

**What happened?**
We have been observing periodic e2e test failures lately. `CreateMultiStargateAndDatacenter` is one of the tests.

When deploying Stargate on its own, i.e., not part of a K8ssandraCluster, and with mulitple racks it can cause schema a ConfigurationException.

I investigated a recent failure. The test failed on a CassandraDatacenter readiness check:

```
Error:     suite_test.go:761: check that datacenter dc1 in cluster kind-k8ssandra-0 is ready
    suite_test.go:763: 
        	Error Trace:	suite_test.go:763
        	            				suite_test.go:530
        	            				suite_test.go:236
        	Error:      	Condition never satisfied
        	Test:       	TestOperator/CreateMultiStargateAndDatacenter
        	Messages:   	timed out waiting for datacenter dc1 to become ready
```

I downloaded the logs and saw this exception in a couple of the Cassadra pods' logs:

```
INFO  [MigrationStage:1] 2022-01-13 17:37:57,798 ColumnFamilyStore.java:432 - Initializing data_endpoint_auth.token
ERROR [MigrationStage:1] 2022-01-13 17:37:59,046 CassandraDaemon.java:244 - Exception in thread Thread[MigrationStage:1,5,main]
org.apache.cassandra.exceptions.ConfigurationException: Column family ID mismatch (found 8c5c36a0-7497-11ec-b2a0-f7a76f46da7d; expected 8bb2fe50-7497-11ec-9837-6fe6aca8f868)
        at org.apache.cassandra.config.CFMetaData.validateCompatibility(CFMetaData.java:1000) ~[apache-cassandra-3.11.11.jar:3.11.11]
        at org.apache.cassandra.config.CFMetaData.apply(CFMetaData.java:953) ~[apache-cassandra-3.11.11.jar:3.11.11]
        at org.apache.cassandra.config.Schema.updateTable(Schema.java:687) ~[apache-cassandra-3.11.11.jar:3.11.11]
        at org.apache.cassandra.schema.SchemaKeyspace.updateKeyspace(SchemaKeyspace.java:1478) ~[apache-cassandra-3.11.11.jar:3.11.11]
        at org.apache.cassandra.schema.SchemaKeyspace.mergeSchema(SchemaKeyspace.java:1434) ~[apache-cassandra-3.11.11.jar:3.11.11]
        at org.apache.cassandra.schema.SchemaKeyspace.mergeSchema(SchemaKeyspace.java:1403) ~[apache-cassandra-3.11.11.jar:3.11.11]
        at org.apache.cassandra.schema.SchemaKeyspace.mergeSchemaAndAnnounceVersion(SchemaKeyspace.java:1380) ~[apache-cassandra-3.11.11.jar:3.11.11]
        at org.apache.cassandra.db.DefinitionsUpdateVerbHandler$1.runMayThrow(DefinitionsUpdateVerbHandler.java:51) ~[apache-cassandra-3.11.11.jar:3.11.11]
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-3.11.11.jar:3.11.11]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_312]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_312]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_312]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_312]
        at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:84) [apache-cassandra-3.11.11.jar:3.11.11]
        at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_312]
```

The same exception is also reported in Stargate's logs. 

When the CassandraDatacenter is configure to use racks, the Stargate controller will, by default, try to spread Stargate instances evenly across racks. It creates a separate Deployment per rack. The problem is that the Stargate instances were started too soon, one after the other. I confirmed this in the k8ssandra-operator logs:

`2022-01-13T17:36:33.945Z        INFO    controller-runtime.manager.controller.stargate  Stargate Deployment not found, creating a new one       {"reconciler group": "stargate.k8ssandra.io", "reconciler kind": "Stargate", "name": "s1", "namespace": "multi-stargate-fcb96g", "Deployment": {"namespace": "multi-stargate-fcb96g", "name": "test-dc1-rack1-stargate-deployment"}}`

`2022-01-13T17:36:34.005Z        INFO    controller-runtime.manager.controller.stargate  Stargate Deployment not found, creating a new one       {"reconciler group": "stargate.k8ssandra.io", "reconciler kind": "Stargate", "name": "s1", "namespace": "multi-stargate-fcb96g", "Deployment": {"namespace": "multi-stargate-fcb96g", "name": "test-dc1-rack2-stargate-deployment"}}`

`2022-01-13T17:36:34.085Z        INFO    controller-runtime.manager.controller.stargate  Stargate Deployment not found, creating a new one       {"reconciler group": "stargate.k8ssandra.io", "reconciler kind": "Stargate", "name": "s1", "namespace": "multi-stargate-fcb96g", "Deployment": {"namespace": "multi-stargate-fcb96g", "name": "test-dc1-rack3-stargate-deployment"}}`

Notice how close together in time `test-dc1-rack2-stargate-deployment` and `test-dc1-rack3-stargate-deployment` are created. We basically have a race condition with respect to initializing. 

This isn't a problem when Stargate is deployed as part of a K8ssandraCluster because the K8ssandraCluster controller initializes the Stargate `data_endpoint_auth` keyspace and the `data_endpoint_auth.token` before deploying Stargate; so, there is no race condition. While deploying Stargate on its own is probably a pretty low priority scenario, we definitely want to fix the flaky tests.

**Did you expect to see something different?**
If Stargate is deployed on its own and not part of a K8ssandraCluster, then the Stargate controller should wait to create the Deployment until the previously created Deployment becomes ready.



┆Issue is synchronized with this [Jira Task](https://k8ssandra.atlassian.net/browse/K8SSAND-1170) by [Unito](https://www.unito.io)
┆friendlyId: K8SSAND-1170
┆priority: Medium


Pull request description:
Reconcile standalone Stargate auth schema
<!--  Thanks for sending a pull request!  Before submitting:

1. Read our CONTRIBUTING.md guide
2. Rebase your PR if it gets out of sync with main
-->

**What this PR does**:

Yet Another Attempt (TM) to fix #297 :-) 

Implements @jsanda suggestion outlined here : https://github.com/k8ssandra/k8ssandra-operator/pull/304#issuecomment-1015073664

**Which issue(s) this PR fixes**:
Fixes #297

**Checklist**
- [ ] Changes manually tested
- [ ] Automated Tests added/updated
- [ ] Documentation added/updated
- [ ] CHANGELOG.md updated (not required for documentation PRs)
- [ ] CLA Signed:  [DataStax CLA](https://cla.datastax.com/)


Patch:
diff --git a/CHANGELOG/CHANGELOG-1.0.md b/CHANGELOG/CHANGELOG-1.0.md
index 86bc70a0..056d4022 100644
--- a/CHANGELOG/CHANGELOG-1.0.md
+++ b/CHANGELOG/CHANGELOG-1.0.md
@@ -26,6 +26,7 @@ When cutting a new release, update the `unreleased` heading to the tag being gen
 * [CHANGE] [#136](https://github.com/k8ssandra/k8ssandra-operator/issues/136) Add shortNames for the K8ssandraCluster CRD
 * [ENHANCEMENT] [#170](https://github.com/k8ssandra/k8ssandra-operator/issues/170) Enforce cluster-wide authentication by default 
 * [ENHANCEMENT] [#267](https://github.com/k8ssandra/k8ssandra-operator/issues/267) Use the K8ssandraCluster name as cluster name for CassandraDatacenter objects
+* [ENHANCEMENT] [#297](https://github.com/k8ssandra/k8ssandra-operator/issues/297) Reconcile standalone Stargate auth schema
 
 ## v1.0.0-alpha.2 - 2021-12-03
 
diff --git a/apis/config/v1beta1/zz_generated.deepcopy.go b/apis/config/v1beta1/zz_generated.deepcopy.go
index f0bc5e66..40a35a45 100644
--- a/apis/config/v1beta1/zz_generated.deepcopy.go
+++ b/apis/config/v1beta1/zz_generated.deepcopy.go
@@ -1,4 +1,3 @@
-//go:build !ignore_autogenerated
 // +build !ignore_autogenerated
 
 /*
diff --git a/apis/k8ssandra/v1alpha1/zz_generated.deepcopy.go b/apis/k8ssandra/v1alpha1/zz_generated.deepcopy.go
index c532fbef..34ff5ed5 100644
--- a/apis/k8ssandra/v1alpha1/zz_generated.deepcopy.go
+++ b/apis/k8ssandra/v1alpha1/zz_generated.deepcopy.go
@@ -1,4 +1,3 @@
-//go:build !ignore_autogenerated
 // +build !ignore_autogenerated
 
 /*
diff --git a/apis/medusa/v1alpha1/zz_generated.deepcopy.go b/apis/medusa/v1alpha1/zz_generated.deepcopy.go
index 25b015dc..a6581894 100644
--- a/apis/medusa/v1alpha1/zz_generated.deepcopy.go
+++ b/apis/medusa/v1alpha1/zz_generated.deepcopy.go
@@ -1,4 +1,3 @@
-//go:build !ignore_autogenerated
 // +build !ignore_autogenerated
 
 /*
diff --git a/apis/reaper/v1alpha1/reaper_types.go b/apis/reaper/v1alpha1/reaper_types.go
index efe0a247..05058840 100644
--- a/apis/reaper/v1alpha1/reaper_types.go
+++ b/apis/reaper/v1alpha1/reaper_types.go
@@ -246,7 +246,8 @@ type ReaperStatus struct {
 
 	// Progress is the progress of this Reaper object.
 	// +kubebuilder:validation:Enum=Pending;Deploying;Configuring;Running
-	Progress ReaperProgress `json:"progress"`
+	// +optional
+	Progress ReaperProgress `json:"progress,omitempty"`
 
 	// +optional
 	Conditions []ReaperCondition `json:"conditions,omitempty"`
diff --git a/apis/reaper/v1alpha1/zz_generated.deepcopy.go b/apis/reaper/v1alpha1/zz_generated.deepcopy.go
index 2536af3b..25db5b00 100644
--- a/apis/reaper/v1alpha1/zz_generated.deepcopy.go
+++ b/apis/reaper/v1alpha1/zz_generated.deepcopy.go
@@ -1,4 +1,3 @@
-//go:build !ignore_autogenerated
 // +build !ignore_autogenerated
 
 /*
diff --git a/apis/replication/v1alpha1/zz_generated.deepcopy.go b/apis/replication/v1alpha1/zz_generated.deepcopy.go
index 710fcead..a397bc19 100644
--- a/apis/replication/v1alpha1/zz_generated.deepcopy.go
+++ b/apis/replication/v1alpha1/zz_generated.deepcopy.go
@@ -1,4 +1,3 @@
-//go:build !ignore_autogenerated
 // +build !ignore_autogenerated
 
 /*
diff --git a/apis/stargate/v1alpha1/stargate_types.go b/apis/stargate/v1alpha1/stargate_types.go
index cf665475..311cbd51 100644
--- a/apis/stargate/v1alpha1/stargate_types.go
+++ b/apis/stargate/v1alpha1/stargate_types.go
@@ -208,7 +208,8 @@ type StargateStatus struct {
 
 	// Progress is the progress of this Stargate object.
 	// +kubebuilder:validation:Enum=Pending;Deploying;Running
-	Progress StargateProgress `json:"progress"`
+	// +optional
+	Progress StargateProgress `json:"progress,omitempty"`
 
 	// +optional
 	Conditions []StargateCondition `json:"conditions,omitempty"`
diff --git a/apis/stargate/v1alpha1/zz_generated.deepcopy.go b/apis/stargate/v1alpha1/zz_generated.deepcopy.go
index 339514bc..5a1561ca 100644
--- a/apis/stargate/v1alpha1/zz_generated.deepcopy.go
+++ b/apis/stargate/v1alpha1/zz_generated.deepcopy.go
@@ -1,4 +1,3 @@
-//go:build !ignore_autogenerated
 // +build !ignore_autogenerated
 
 /*
diff --git a/apis/telemetry/v1alpha1/zz_generated.deepcopy.go b/apis/telemetry/v1alpha1/zz_generated.deepcopy.go
index 6eb70e8d..7be31b4a 100644
--- a/apis/telemetry/v1alpha1/zz_generated.deepcopy.go
+++ b/apis/telemetry/v1alpha1/zz_generated.deepcopy.go
@@ -1,4 +1,3 @@
-//go:build !ignore_autogenerated
 // +build !ignore_autogenerated
 
 /*
diff --git a/config/crd/bases/k8ssandra.io_k8ssandraclusters.yaml b/config/crd/bases/k8ssandra.io_k8ssandraclusters.yaml
index 13d95a0a..8ecd9d12 100644
--- a/config/crd/bases/k8ssandra.io_k8ssandraclusters.yaml
+++ b/config/crd/bases/k8ssandra.io_k8ssandraclusters.yaml
@@ -11875,8 +11875,6 @@ spec:
                           - Configuring
                           - Running
                           type: string
-                      required:
-                      - progress
                       type: object
                     stargate:
                       description: StargateStatus defines the observed state of a
@@ -11949,7 +11947,6 @@ spec:
                           type: integer
                       required:
                       - availableReplicas
-                      - progress
                       - readyReplicas
                       - replicas
                       - updatedReplicas
diff --git a/config/crd/bases/reaper.k8ssandra.io_reapers.yaml b/config/crd/bases/reaper.k8ssandra.io_reapers.yaml
index e503de1f..c8a866ac 100644
--- a/config/crd/bases/reaper.k8ssandra.io_reapers.yaml
+++ b/config/crd/bases/reaper.k8ssandra.io_reapers.yaml
@@ -1886,8 +1886,6 @@ spec:
                 - Configuring
                 - Running
                 type: string
-            required:
-            - progress
             type: object
         type: object
     served: true
diff --git a/config/crd/bases/stargate.k8ssandra.io_stargates.yaml b/config/crd/bases/stargate.k8ssandra.io_stargates.yaml
index ffdf33aa..b3c72c86 100644
--- a/config/crd/bases/stargate.k8ssandra.io_stargates.yaml
+++ b/config/crd/bases/stargate.k8ssandra.io_stargates.yaml
@@ -2758,7 +2758,6 @@ spec:
                 type: integer
             required:
             - availableReplicas
-            - progress
             - readyReplicas
             - replicas
             - updatedReplicas
diff --git a/controllers/k8ssandra/cleanup.go b/controllers/k8ssandra/cleanup.go
index f2587215..d27a9081 100644
--- a/controllers/k8ssandra/cleanup.go
+++ b/controllers/k8ssandra/cleanup.go
@@ -60,7 +60,7 @@ func (r *K8ssandraClusterReconciler) checkDeletion(ctx context.Context, kc *api.
 			hasErrors = true
 		}
 
-		selector := k8ssandralabels.CreatedByK8ssandraControllerLabels(kcKey)
+		selector := k8ssandralabels.PartOfLabels(kcKey)
 		stargateList := &stargateapi.StargateList{}
 		options := client.ListOptions{
 			Namespace:     namespace,
diff --git a/controllers/k8ssandra/reaper.go b/controllers/k8ssandra/reaper.go
index d37ebdcf..34c618bb 100644
--- a/controllers/k8ssandra/reaper.go
+++ b/controllers/k8ssandra/reaper.go
@@ -65,7 +65,7 @@ func (r *K8ssandraClusterReconciler) reconcileReaperSchema(
 
 		err = managementApiFacade.EnsureKeyspaceReplication(
 			keyspace,
-			cassandra.ComputeReplication(3, kc.Spec.Cassandra.Datacenters...),
+			cassandra.ComputeReplication(3, dcs...),
 		)
 		if err != nil {
 			logger.Error(err, "Failed to ensure keyspace replication")
@@ -154,7 +154,7 @@ func (r *K8ssandraClusterReconciler) reconcileReaper(
 				logger.Error(err, "Failed to get Reaper resource")
 				return result.Error(err)
 			}
-		} else if k8ssandralabels.IsCreatedByK8ssandraController(actualReaper, kcKey) {
+		} else if k8ssandralabels.IsPartOf(actualReaper, kcKey) {
 			if err = remoteClient.Delete(ctx, actualReaper); err != nil {
 				logger.Error(err, "Failed to delete Reaper resource")
 				return result.Error(err)
@@ -177,7 +177,7 @@ func (r *K8ssandraClusterReconciler) deleteReapers(
 	remoteClient client.Client,
 	kcLogger logr.Logger,
 ) (hasErrors bool) {
-	selector := k8ssandralabels.CreatedByK8ssandraControllerLabels(client.ObjectKey{Namespace: kc.Namespace, Name: kc.Name})
+	selector := k8ssandralabels.PartOfLabels(client.ObjectKey{Namespace: kc.Namespace, Name: kc.Name})
 	reaperList := &reaperapi.ReaperList{}
 	options := client.ListOptions{
 		Namespace:     namespace,
diff --git a/controllers/k8ssandra/replication.go b/controllers/k8ssandra/replication.go
index 082e3b10..229ae086 100644
--- a/controllers/k8ssandra/replication.go
+++ b/controllers/k8ssandra/replication.go
@@ -58,7 +58,7 @@ func (r *K8ssandraClusterReconciler) updateReplicationOfSystemKeyspaces(ctx cont
 	}
 
 	keyspaces := []string{"system_traces", "system_distributed", "system_auth"}
-	replication := cassandra.ComputeReplication(3, kc.Spec.Cassandra.Datacenters...)
+	replication := cassandra.ComputeReplicationFromDcTemplates(3, kc.Spec.Cassandra.Datacenters...)
 
 	logger.Info("Preparing to update replication for system keyspaces", "replication", replication)
 
diff --git a/controllers/k8ssandra/secrets.go b/controllers/k8ssandra/secrets.go
index c860585c..676fbc65 100644
--- a/controllers/k8ssandra/secrets.go
+++ b/controllers/k8ssandra/secrets.go
@@ -8,6 +8,7 @@ import (
 	"github.com/k8ssandra/k8ssandra-operator/pkg/result"
 	"github.com/k8ssandra/k8ssandra-operator/pkg/secret"
 	"github.com/k8ssandra/k8ssandra-operator/pkg/utils"
+	corev1 "k8s.io/api/core/v1"
 )
 
 func (r *K8ssandraClusterReconciler) reconcileSuperuserSecret(ctx context.Context, kc *api.K8ssandraCluster, logger logr.Logger) result.ReconcileResult {
@@ -34,15 +35,19 @@ func (r *K8ssandraClusterReconciler) reconcileSuperuserSecret(ctx context.Contex
 }
 
 func (r *K8ssandraClusterReconciler) reconcileReaperSecrets(ctx context.Context, kc *api.K8ssandraCluster, logger logr.Logger) result.ReconcileResult {
-	if kc.Spec.Reaper != nil {
+	if kc.HasReapers() {
 		// Reaper secrets are only required when authentication is enabled on the cluster
 		if kc.Spec.IsAuthEnabled() {
 			logger.Info("Reconciling Reaper user secrets")
-			cassandraUserSecretRef := kc.Spec.Reaper.CassandraUserSecretRef
+			var cassandraUserSecretRef corev1.LocalObjectReference
+			var jmxUserSecretRef corev1.LocalObjectReference
+			if kc.Spec.Reaper != nil {
+				cassandraUserSecretRef = kc.Spec.Reaper.CassandraUserSecretRef
+				jmxUserSecretRef = kc.Spec.Reaper.JmxUserSecretRef
+			}
 			if cassandraUserSecretRef.Name == "" {
 				cassandraUserSecretRef.Name = reaper.DefaultUserSecretName(kc.Name)
 			}
-			jmxUserSecretRef := kc.Spec.Reaper.JmxUserSecretRef
 			if jmxUserSecretRef.Name == "" {
 				jmxUserSecretRef.Name = reaper.DefaultJmxUserSecretName(kc.Name)
 			}
diff --git a/controllers/k8ssandra/stargate.go b/controllers/k8ssandra/stargate.go
index 73fbd713..ea06a57e 100644
--- a/controllers/k8ssandra/stargate.go
+++ b/controllers/k8ssandra/stargate.go
@@ -7,7 +7,6 @@ import (
 	api "github.com/k8ssandra/k8ssandra-operator/apis/k8ssandra/v1alpha1"
 	stargateapi "github.com/k8ssandra/k8ssandra-operator/apis/stargate/v1alpha1"
 	"github.com/k8ssandra/k8ssandra-operator/pkg/annotations"
-	"github.com/k8ssandra/k8ssandra-operator/pkg/cassandra"
 	"github.com/k8ssandra/k8ssandra-operator/pkg/labels"
 	"github.com/k8ssandra/k8ssandra-operator/pkg/result"
 	"github.com/k8ssandra/k8ssandra-operator/pkg/stargate"
@@ -89,7 +88,7 @@ func (r *K8ssandraClusterReconciler) reconcileStargate(
 				logger.Error(err, "Failed to get Stargate")
 				return result.Error(err)
 			}
-		} else if labels.IsCreatedByK8ssandraController(actualStargate, kcKey) {
+		} else if labels.IsPartOf(actualStargate, kcKey) {
 			if err := remoteClient.Delete(ctx, actualStargate); err != nil {
 				logger.Error(err, "Failed to delete Stargate")
 				return result.Error(err)
@@ -155,12 +154,16 @@ func (r *K8ssandraClusterReconciler) setStatusForStargate(kc *api.K8ssandraClust
 	return nil
 }
 
-func (r *K8ssandraClusterReconciler) reconcileStargateAuthSchema(ctx context.Context, kc *api.K8ssandraCluster, dcs []*cassdcapi.CassandraDatacenter, logger logr.Logger) result.ReconcileResult {
+func (r *K8ssandraClusterReconciler) reconcileStargateAuthSchema(
+	ctx context.Context,
+	kc *api.K8ssandraCluster,
+	dcs []*cassdcapi.CassandraDatacenter,
+	logger logr.Logger,
+) result.ReconcileResult {
 	if !kc.HasStargates() {
 		return result.Continue()
 	}
 
-	logger.Info("Reconciling Stargate auth schema")
 	dcTemplate := kc.Spec.Cassandra.Datacenters[0]
 
 	if remoteClient, err := r.ClientCache.GetRemoteClient(dcTemplate.K8sContext); err != nil {
@@ -173,21 +176,11 @@ func (r *K8ssandraClusterReconciler) reconcileStargateAuthSchema(ctx context.Con
 			logger.Error(err, "Failed to create ManagementApiFacade")
 			return result.Error(err)
 		}
-
-		replication := cassandra.ComputeReplication(3, kc.Spec.Cassandra.Datacenters...)
-		if err = managementApi.EnsureKeyspaceReplication(stargate.AuthKeyspace, replication); err != nil {
-			logger.Error(err, "Failed to ensure keyspace replication")
-			return result.Error(err)
-		}
-
-		if err = stargate.ReconcileAuthTable(managementApi, logger); err != nil {
-			logger.Error(err, "Failed to reconcile Stargate auth table")
+		if err = stargate.ReconcileAuthKeyspace(dcs, managementApi, logger); err != nil {
 			return result.Error(err)
 		}
-
 		return result.Continue()
 	}
-
 }
 
 func (r *K8ssandraClusterReconciler) removeStargateStatus(kc *api.K8ssandraCluster, dcName string) {
diff --git a/controllers/stargate/stargate_controller.go b/controllers/stargate/stargate_controller.go
index 9a0f5760..45a00b0a 100644
--- a/controllers/stargate/stargate_controller.go
+++ b/controllers/stargate/stargate_controller.go
@@ -19,11 +19,11 @@ package stargate
 import (
 	"context"
 	"fmt"
-
 	cassdcapi "github.com/k8ssandra/cass-operator/apis/cassandra/v1beta1"
 	"github.com/k8ssandra/k8ssandra-operator/pkg/annotations"
 	"github.com/k8ssandra/k8ssandra-operator/pkg/cassandra"
 	"github.com/k8ssandra/k8ssandra-operator/pkg/config"
+	"github.com/k8ssandra/k8ssandra-operator/pkg/labels"
 	stargateutil "github.com/k8ssandra/k8ssandra-operator/pkg/stargate"
 	appsv1 "k8s.io/api/apps/v1"
 	corev1 "k8s.io/api/core/v1"
@@ -51,7 +51,8 @@ import (
 type StargateReconciler struct {
 	*config.ReconcilerConfig
 	client.Client
-	Scheme *runtime.Scheme
+	Scheme        *runtime.Scheme
+	ManagementApi cassandra.ManagementApiFactory
 }
 
 func (r *StargateReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
@@ -144,6 +145,17 @@ func (r *StargateReconciler) Reconcile(ctx context.Context, req ctrl.Request) (c
 			"Stargate", req.NamespacedName)
 	}
 
+	if !labels.IsOwnedByK8ssandraController(stargate) {
+		logger.Info("Stargate resource is standalone, reconciling auth schema now")
+		managementApi, err := r.ManagementApi.NewManagementApiFacade(ctx, actualDc, r.Client, logger)
+		if err != nil {
+			logger.Error(err, "Failed to create ManagementApiFacade")
+			return ctrl.Result{}, err
+		} else if err = stargateutil.ReconcileAuthKeyspace([]*cassdcapi.CassandraDatacenter{actualDc}, managementApi, logger); err != nil {
+			return ctrl.Result{}, err
+		}
+	}
+
 	// Compute the desired deployments
 	desiredDeployments := stargateutil.NewDeployments(stargate, actualDc)
 
diff --git a/main.go b/main.go
index e0e35471..4db66025 100644
--- a/main.go
+++ b/main.go
@@ -75,7 +75,7 @@ func init() {
 	utilruntime.Must(reaperapi.AddToScheme(scheme))
 	utilruntime.Must(promapi.AddToScheme(scheme))
 	utilruntime.Must(medusav1alpha1.AddToScheme(scheme))
-	//+kubebuilder:scaffold:scheme
+	// +kubebuilder:scaffold:scheme
 }
 
 func main() {
@@ -218,6 +218,7 @@ func main() {
 		ReconcilerConfig: reconcilerConfig,
 		Client:           mgr.GetClient(),
 		Scheme:           mgr.GetScheme(),
+		ManagementApi:    cassandra.NewManagementApiFactory(),
 	}).SetupWithManager(mgr); err != nil {
 		setupLog.Error(err, "unable to create controller", "controller", "Stargate")
 		os.Exit(1)
@@ -250,7 +251,7 @@ func main() {
 		setupLog.Error(err, "unable to create controller", "controller", "CassandraRestore")
 		os.Exit(1)
 	}
-	//+kubebuilder:scaffold:builder
+	// +kubebuilder:scaffold:builder
 
 	if err := mgr.AddHealthzCheck("healthz", healthz.Ping); err != nil {
 		setupLog.Error(err, "unable to set up health check")
diff --git a/pkg/cassandra/util.go b/pkg/cassandra/util.go
index e8711677..35507d30 100644
--- a/pkg/cassandra/util.go
+++ b/pkg/cassandra/util.go
@@ -45,7 +45,19 @@ func ComputeSystemReplication(kluster *api.K8ssandraCluster) SystemReplication {
 	return SystemReplication{Datacenters: dcNames, ReplicationFactor: int(rf)}
 }
 
-func ComputeReplication(maxReplicationPerDc int, datacenters ...api.CassandraDatacenterTemplate) map[string]int {
+// ComputeReplication computes the desired replication for each dc, taking into account the desired maximum replication
+// per dc.
+func ComputeReplication(maxReplicationPerDc int, datacenters ...*cassdcapi.CassandraDatacenter) map[string]int {
+	desiredReplication := make(map[string]int, len(datacenters))
+	for _, dcTemplate := range datacenters {
+		replicationFactor := int(math.Min(float64(maxReplicationPerDc), float64(dcTemplate.Spec.Size)))
+		desiredReplication[dcTemplate.Name] = replicationFactor
+	}
+	return desiredReplication
+}
+
+// ComputeReplicationFromDcTemplates is similar to ComputeReplication but takes dc templates as parameters.
+func ComputeReplicationFromDcTemplates(maxReplicationPerDc int, datacenters ...api.CassandraDatacenterTemplate) map[string]int {
 	desiredReplication := make(map[string]int, len(datacenters))
 	for _, dcTemplate := range datacenters {
 		replicationFactor := int(math.Min(float64(maxReplicationPerDc), float64(dcTemplate.Size)))
diff --git a/pkg/images/zz_generated.deepcopy.go b/pkg/images/zz_generated.deepcopy.go
index ba9c8d29..10c3b6c1 100644
--- a/pkg/images/zz_generated.deepcopy.go
+++ b/pkg/images/zz_generated.deepcopy.go
@@ -1,4 +1,3 @@
-//go:build !ignore_autogenerated
 // +build !ignore_autogenerated
 
 /*
diff --git a/pkg/labels/labels.go b/pkg/labels/labels.go
index d14a6796..ce55fdc9 100644
--- a/pkg/labels/labels.go
+++ b/pkg/labels/labels.go
@@ -24,6 +24,10 @@ func GetLabel(component Labeled, labelKey string) string {
 	return m[labelKey]
 }
 
+func HasLabel(component Labeled, labelKey string) bool {
+	return GetLabel(component, labelKey) != ""
+}
+
 func HasLabelWithValue(component Labeled, labelKey string, labelValue string) bool {
 	return GetLabel(component, labelKey) == labelValue
 }
@@ -54,22 +58,30 @@ func ManagedByLabels(klusterKey client.ObjectKey) map[string]string {
 	}
 }
 
-// IsCreatedByK8ssandraController returns true if this component was created by the k8ssandra-cluster controller, and
-// belongs to the K8ssandraCluster resource specified by klusterKey. klusterKey referns to the namespace and
-// name of the K8ssandraCluster.
-func IsCreatedByK8ssandraController(component Labeled, klusterKey client.ObjectKey) bool {
+// IsPartOf returns true if this component was created by the k8ssandra-cluster controller, and belongs to the
+// K8ssandraCluster resource specified by klusterKey. klusterKey refers to the namespace and name of the
+// K8ssandraCluster.
+func IsPartOf(component Labeled, klusterKey client.ObjectKey) bool {
 	return HasLabelWithValue(component, k8ssandraapi.CreatedByLabel, k8ssandraapi.CreatedByLabelValueK8ssandraClusterController) &&
 		HasLabelWithValue(component, k8ssandraapi.K8ssandraClusterNameLabel, klusterKey.Name) &&
 		HasLabelWithValue(component, k8ssandraapi.K8ssandraClusterNamespaceLabel, klusterKey.Namespace)
 }
 
-// CreatedByK8ssandraControllerLabels returns the labels used to identify a component created by the k8ssandra-cluster
-// controller, and belonging to the K8ssandraCluster resource specified by klusterKey, which is namespace and name
-// of the K8ssandraCluster.
-func CreatedByK8ssandraControllerLabels(klusterKey client.ObjectKey) map[string]string {
+// PartOfLabels returns the labels used to identify a component created by the k8ssandra-cluster controller, and
+// belonging to the K8ssandraCluster resource specified by klusterKey, which is namespace and name of the
+// K8ssandraCluster.
+func PartOfLabels(klusterKey client.ObjectKey) map[string]string {
+	// TODO add k8ssandraapi.PartOfLabel entry here, already done for telemetry objects elsewhere
 	return map[string]string{
 		k8ssandraapi.CreatedByLabel:                 k8ssandraapi.CreatedByLabelValueK8ssandraClusterController,
 		k8ssandraapi.K8ssandraClusterNameLabel:      klusterKey.Name,
 		k8ssandraapi.K8ssandraClusterNamespaceLabel: klusterKey.Namespace,
 	}
 }
+
+// IsOwnedByK8ssandraController returns true if this component was created by the k8ssandra-cluster controller.
+func IsOwnedByK8ssandraController(component Labeled) bool {
+	return HasLabelWithValue(component, k8ssandraapi.CreatedByLabel, k8ssandraapi.CreatedByLabelValueK8ssandraClusterController) &&
+		HasLabel(component, k8ssandraapi.K8ssandraClusterNameLabel) &&
+		HasLabel(component, k8ssandraapi.K8ssandraClusterNamespaceLabel)
+}
diff --git a/pkg/stargate/auth_schema.go b/pkg/stargate/auth_schema.go
index a15102ef..437d4182 100644
--- a/pkg/stargate/auth_schema.go
+++ b/pkg/stargate/auth_schema.go
@@ -3,6 +3,7 @@ package stargate
 import (
 	"fmt"
 	"github.com/go-logr/logr"
+	cassdcapi "github.com/k8ssandra/cass-operator/apis/cassandra/v1beta1"
 	"github.com/k8ssandra/cass-operator/pkg/httphelper"
 	"github.com/k8ssandra/k8ssandra-operator/pkg/cassandra"
 )
@@ -22,23 +23,37 @@ var authTableDefinition = &httphelper.TableDefinition{
 	},
 }
 
+// ReconcileAuthKeyspace ensures that the Stargate auth schema exists, has the appropriate replication, and contains the
+// appropriate tables.
+func ReconcileAuthKeyspace(dcs []*cassdcapi.CassandraDatacenter, managementApi cassandra.ManagementApiFacade, logger logr.Logger) error {
+	replication := cassandra.ComputeReplication(3, dcs...)
+	logger.Info(fmt.Sprintf("Reconciling Stargate auth keyspace %v", AuthKeyspace))
+	if err := managementApi.EnsureKeyspaceReplication(AuthKeyspace, replication); err != nil {
+		logger.Error(err, "Failed to ensure keyspace replication")
+		return err
+	}
+	return ReconcileAuthTable(managementApi, logger)
+}
+
 // ReconcileAuthTable ensures that the Stargate auth schema contains the appropriate tables. Note that we don't do much
 // here and currently, we only check if the token table exists, without checking if the table definition matches the
 // expected one. If the auth schema becomes more complex in the future, we'd need to find a more robust solution, maybe
 // à la Reaper.
 func ReconcileAuthTable(managementApi cassandra.ManagementApiFacade, logger logr.Logger) error {
+	logger.Info(fmt.Sprintf("Reconciling Stargate auth table %v.%v", AuthKeyspace, AuthTable))
 	if tables, err := managementApi.ListTables(AuthKeyspace); err != nil {
 		return err
 	} else if len(tables) == 0 {
-		logger.Info(fmt.Sprintf("table %s does not exist in keyspace %v, creating it", AuthTable, AuthKeyspace))
+		logger.Info(fmt.Sprintf("Table %s does not exist in keyspace %v, creating it", AuthTable, AuthKeyspace))
 		if err := managementApi.CreateTable(authTableDefinition); err != nil {
+			logger.Error(err, fmt.Sprintf("Failed to create table %v", AuthTable))
 			return err
 		} else {
 			logger.Info(fmt.Sprintf("Table %s successfully created", AuthTable))
 			return nil
 		}
 	} else {
-		logger.Info(fmt.Sprintf("table %s already exists in keyspace %v", AuthTable, AuthKeyspace))
+		logger.Info(fmt.Sprintf("Table %v.%s already exists", AuthKeyspace, AuthTable))
 		return nil
 	}
 }
diff --git a/pkg/telemetry/prometheus_resourcer.go b/pkg/telemetry/prometheus_resourcer.go
index f60adb37..0a9dd0d8 100644
--- a/pkg/telemetry/prometheus_resourcer.go
+++ b/pkg/telemetry/prometheus_resourcer.go
@@ -4,7 +4,7 @@ package telemetry
 
 import (
 	"context"
-	"errors"
+
 	"github.com/go-logr/logr"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 	runtimeclient "sigs.k8s.io/controller-runtime/pkg/client"
@@ -86,21 +86,17 @@ func (cfg PrometheusResourcer) UpdateResources(
 }
 
 // CleanupResources executes the cleanup of any resources on the cluster, once they are no longer required.
-// CleanupResources does not rely on user-defined CommonLabels to select which resources to delete. It uses only
-// k8ssandra-operator defined labels.
 func (cfg PrometheusResourcer) CleanupResources(ctx context.Context, client runtimeclient.Client) error {
-	if len(cfg.CommonLabels) < 1 {
-		return errors.New("no labels were found to target deletion request")
-	}
-	if err := cfg.validate(); err != nil {
-		return TelemetryConfigIncomplete{}
+	targetSM := promapi.ServiceMonitor{}
+	err := client.Get(ctx, types.NamespacedName{Name: cfg.ServiceMonitorName, Namespace: cfg.MonitoringTargetNS}, &targetSM)
+	if err != nil {
+		if k8serrors.IsNotFound(err) {
+			return nil
+		} else {
+			return err
+		}
 	}
-	var deleteTargets promapi.ServiceMonitor
-	err := client.DeleteAllOf(ctx,
-		&deleteTargets,
-		runtimeclient.InNamespace(cfg.MonitoringTargetNS),
-		runtimeclient.MatchingLabels(cfg.CommonLabels),
-	)
+	err = client.Delete(ctx, &targetSM)
 	if err != nil {
 		return err
 	}


Test patch:
diff --git a/controllers/stargate/stargate_controller_test.go b/controllers/stargate/stargate_controller_test.go
index 697a0b2f..5ec23ef9 100644
--- a/controllers/stargate/stargate_controller_test.go
+++ b/controllers/stargate/stargate_controller_test.go
@@ -2,9 +2,14 @@ package stargate
 
 import (
 	"context"
+	"github.com/go-logr/logr"
+	"github.com/k8ssandra/cass-operator/pkg/httphelper"
 	telemetryapi "github.com/k8ssandra/k8ssandra-operator/apis/telemetry/v1alpha1"
+	"github.com/k8ssandra/k8ssandra-operator/pkg/cassandra"
+	"github.com/k8ssandra/k8ssandra-operator/pkg/mocks"
 	"github.com/k8ssandra/k8ssandra-operator/pkg/stargate"
 	promapi "github.com/prometheus-operator/prometheus-operator/pkg/apis/monitoring/v1"
+	"github.com/stretchr/testify/mock"
 	k8serrors "k8s.io/apimachinery/pkg/api/errors"
 	"testing"
 	"time"
@@ -32,6 +37,8 @@ const (
 	interval = time.Millisecond * 500
 )
 
+var managementApi = &fakeManagementApiFactory{}
+
 func TestStargate(t *testing.T) {
 	ctx := testutils.TestSetup(t)
 	ctx, cancel := context.WithCancel(ctx)
@@ -41,6 +48,7 @@ func TestStargate(t *testing.T) {
 			ReconcilerConfig: config.InitConfig(),
 			Client:           mgr.GetClient(),
 			Scheme:           scheme.Scheme,
+			ManagementApi:    managementApi,
 		}).SetupWithManager(mgr)
 		return err
 	})
@@ -406,3 +414,16 @@ func testCreateStargateMultiRack(t *testing.T, testClient client.Client) {
 	assert.Equal(t, corev1.ConditionTrue, stargate.Status.Conditions[0].Status)
 
 }
+
+type fakeManagementApiFactory struct {
+}
+
+func (f fakeManagementApiFactory) NewManagementApiFacade(context.Context, *cassdcapi.CassandraDatacenter, client.Client, logr.Logger) (cassandra.ManagementApiFacade, error) {
+	m := new(mocks.ManagementApiFacade)
+	m.On("EnsureKeyspaceReplication", mock.Anything, mock.Anything).Return(nil)
+	m.On("ListTables", stargate.AuthKeyspace).Return([]string{"token"}, nil)
+	m.On("CreateTable", mock.MatchedBy(func(def *httphelper.TableDefinition) bool {
+		return def.KeyspaceName == stargate.AuthKeyspace && def.TableName == stargate.AuthTable
+	})).Return(nil)
+	return m, nil
+}
diff --git a/pkg/cassandra/util_test.go b/pkg/cassandra/util_test.go
index 8a03e433..f95a1194 100644
--- a/pkg/cassandra/util_test.go
+++ b/pkg/cassandra/util_test.go
@@ -1,7 +1,9 @@
 package cassandra
 
 import (
+	cassdcapi "github.com/k8ssandra/cass-operator/apis/cassandra/v1beta1"
 	"github.com/stretchr/testify/assert"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 	"testing"
 
 	api "github.com/k8ssandra/k8ssandra-operator/apis/k8ssandra/v1alpha1"
@@ -104,6 +106,69 @@ func TestComputeSystemReplication(t *testing.T) {
 }
 
 func TestComputeReplication(t *testing.T) {
+	tests := []struct {
+		name     string
+		dcs      []*cassdcapi.CassandraDatacenter
+		expected map[string]int
+	}{
+		{
+			"one dc",
+			[]*cassdcapi.CassandraDatacenter{
+				{
+					ObjectMeta: metav1.ObjectMeta{Name: "dc1"},
+					Spec:       cassdcapi.CassandraDatacenterSpec{Size: 3},
+				},
+			},
+			map[string]int{"dc1": 3},
+		},
+		{
+			"small dc",
+			[]*cassdcapi.CassandraDatacenter{
+				{
+					ObjectMeta: metav1.ObjectMeta{Name: "dc1"},
+					Spec:       cassdcapi.CassandraDatacenterSpec{Size: 1},
+				},
+			},
+			map[string]int{"dc1": 1},
+		},
+		{
+			"large dc",
+			[]*cassdcapi.CassandraDatacenter{
+				{
+					ObjectMeta: metav1.ObjectMeta{Name: "dc1"},
+					Spec:       cassdcapi.CassandraDatacenterSpec{Size: 10},
+				},
+			},
+			map[string]int{"dc1": 3},
+		},
+		{
+			"many dcs",
+			[]*cassdcapi.CassandraDatacenter{
+				{
+					ObjectMeta: metav1.ObjectMeta{Name: "dc1"},
+					Spec:       cassdcapi.CassandraDatacenterSpec{Size: 3},
+				},
+				{
+					ObjectMeta: metav1.ObjectMeta{Name: "dc2"},
+					Spec:       cassdcapi.CassandraDatacenterSpec{Size: 1},
+				},
+				{
+					ObjectMeta: metav1.ObjectMeta{Name: "dc3"},
+					Spec:       cassdcapi.CassandraDatacenterSpec{Size: 10},
+				},
+			},
+			map[string]int{"dc1": 3, "dc2": 1, "dc3": 3},
+		},
+	}
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			actual := ComputeReplication(3, tt.dcs...)
+			assert.Equal(t, tt.expected, actual)
+		})
+	}
+}
+
+func TestComputeReplicationFromDcTemplates(t *testing.T) {
 	tests := []struct {
 		name     string
 		dcs      []api.CassandraDatacenterTemplate
@@ -126,7 +191,7 @@ func TestComputeReplication(t *testing.T) {
 	}
 	for _, tt := range tests {
 		t.Run(tt.name, func(t *testing.T) {
-			actual := ComputeReplication(3, tt.dcs...)
+			actual := ComputeReplicationFromDcTemplates(3, tt.dcs...)
 			assert.Equal(t, tt.expected, actual)
 		})
 	}