Repository: cockroachdb/cockroach-operator

Issue:
operator incorrectly uses `--logtostderr` + can't use new --log flag with Operator
I attempted to specify the new [`--log` flag](https://www.cockroachlabs.com/docs/v21.1/configure-logs#flag) in the custom resource:

```
  additionalArgs:
    - "--log='sinks: {file-groups: {ops: {channels: [OPS, HEALTH]}}}'"
```

This resulted in the pods crashing:

```
NAME                                 READY   STATUS             RESTARTS   AGE
cockroach-operator-d4df75996-pdpmd   1/1     Running            0          135m
cockroachdb-0                        0/1     CrashLoopBackOff   11         34m
cockroachdb-1                        0/1     CrashLoopBackOff   11         34m
cockroachdb-2                        0/1     CrashLoopBackOff   11         34m
```

The kubectl logs show:

```
++ expr 8192 / 4
++ expr 8192 / 4
+ exec /cockroach/cockroach.sh start --join=cockroachdb-0.cockroachdb.default:26258,cockroachdb-1.cockroachdb.default:26258,cockroachdb-2.cockroachdb.default:26258 --advertise-host=cockroachdb-0.cockroachdb.default --logtostderr=INFO --certs-dir=/cockroach/cockroach-certs/ --http-port=8080 --sql-addr=:26257 --listen-addr=:26258 --cache 2048MiB --max-sql-memory 2048MiB '--log=sinks: {file-groups: {ops: {channels: [OPS, HEALTH]}}}'
Flag --logtostderr has been deprecated, use --log instead to specify 'sinks: {stderr: {filter: ...}}'.
E210621 20:27:35.192687 1 1@cli/error.go:399  [-] 1  ERROR: --log is incompatible with legacy discrete logging flags
ERROR: --log is incompatible with legacy discrete logging flags
```

Since `--logtostderr` is deprecated in v21.1 (and according to @knz we shouldn't be logging to stderr anyway), perhaps this [line](https://github.com/cockroachdb/cockroach-operator/blob/master/pkg/resource/statefulset.go#L340) in the code should be removed. I believe the logging behavior would then default to [this configuration](https://www.cockroachlabs.com/docs/v21.1/configure-logs#default-logging-configuration).

If we do need to log to `stderr` on Kubernetes, I believe the equivalent v21.1 flag is:

```
--log="sinks: {stderr: {channels: all, filter: INFO}}"
```

It looks like `--logtostderr` is also used in a couple `.golden` files, but I'm not sure how relevant those are.

Hints from issue comments:
@knz: The flag `--logtostderr` should not be used in our reference kubernetes configs.

In fact, we should never recommend logging to stderr. So even the proposed `--log` config would not be desirable. The idea to force logging to stderr is a system design mistake.


@knz: I think the proper recommendation to move forward is as follows:

- remove `--logtostderr`  from all the operator configs
- don't re-add stderr logging with `--log`
- instead, add a config that does the follow:
  - retains a *small* amount of logs (`max-group-size: 1MB`) on ephemeral pod storage
  - sends the bulk of the logs to a network collector
- make the operator instantiate a log collector alongside the crdb pods@chrislovecnm: Is this a version related issue? Are we going to have problems with different supported versions of crdb?@chrislovecnm: https://github.com/cockroachdb/cockroach-operator/blob/8e7c5b6b961e5502692aa7a29289b19d88196696/pkg/resource/statefulset.go#L340

This where we are setting logging.@knz: The fact `--logtostderr` is being used *at all* is an issue that exists for all crdb versions. This should not have been used from the start.

Additionally, starting in v21.1, if the user wants to customize their startup flags with a custom `--log`, that is currently made impossible due to the orchestration currently using `--logtostderr`.@taroface: cc @udnay for visibility@chrislovecnm: I need some requirements.

- what do we have for default?
- we need an api value to do specific configurations@knz: > what do we have for default?

The default config is to write logging output to files `cockroach-xxx.log` in the `logs` sub-directory of the data directory. This is good and exactly what we need for troubleshooting.

There is no operational nor troubleshooting value for production cluster provided by stderr output via `--logtostderr`.

Therefore, the requirement is to remove this flag from the config.

> we need an api value to do specific configurations

The `--log` flag accepts a YAML payload (or use `--log-config-file` to point to a *file* containing a YAML payload). An operator should be able to provide a custom YAML payload in such a way that the flag is passed to crdb only if the custom YAML payload was provided.

@chrislovecnm: Yah the default configuration is funky for k8s. We always need logging to console. So we need output to console and that is why I think my predecessor used that flag.

We need to use --log, but putting yaml in a yaml file might be fun.... 🤔 - have not done that before.

This brings up a good point, the users can break logging pretty easy. CRDB is set up to log as a binary not running inside of a container; log to a file. Do we want to check that a user is not logging to a file or is at least always logging to console and possibly another place?@chrislovecnm: @piyush-singh can you please document business requirements. And keep in mind we always need logging to console.@knz: > at least always logging to console and possibly another place?

I'm saying **not** log to console by default.

If something gets logged to the console, make it only the OPS channel.@chrislovecnm: @knz yah I got you, but how does kubectl log work?@knz: the mistake you're making is assuming we want to see crdb logs in the output of `kubectl log`.
That is incorrect. The useful data we need to extract for troubleshooting is not present in the crdb logs.

It's good to see the *default* stderr output (which is just a server startup message) in the output of `kubectl log`. we *might* have some interest in seeing the output of the OPS logging channel (new in 21.1) via `kubectl log` but I don't know that we have evidence for that (yet).

Whatever additional detailed logging produced by crdb should not be available via `kubectl log`. The logging is largely structured and inadequate for channeling through kube's simplistic logging output. We should instead require an operator to run `cockroach debug zip` or some other related command to extract operational details.@udnay: @knz how does `cockroach debug zip` work? Will it only zip up logs in a specific directory? On k8s the norm is to log app data to stdout as logs in a pod are ephemeral unless the log dir is mounted directly from the host.

Without exec'ing into a pod, how would the operator see the logs if they bypass k8s logging facility?@chrislovecnm: @udnay yah we would need to do a host mount or pvc to make the logs non ephemeral if we want to do some complex logging.

My thoughts are to keep it simple meeting the k8s logging standard.  To your point k8s users are going to expect logs going to stdout, or really we break how logging works in k8s. All of the hosted k8s solutions have log aggregation that rely on console logging.

Yes we may have some edge cases for disk logging, but we need a simple solution first.

I think the “debug zip” is another problem that we need to tackle as well, we may have an issue open already. IMO let’s use this issue for talking about logging and open a different issue for debug.@knz: ok so this is still a "bug" issue, not "enhancement".

I think we've been talking past each other and somehow I have not been successful at conveying how serious the situation is.

I'm going to rephrase this more clearly.

As of today, the current operator code wrt logging has been misdesigned. What is being done was incorrect from the start, and the fact nobody noticed does not detract from the fact it's a problem and needs to be corrected. (See [this definition](https://www.webopedia.com/definitions/schroedinbug/))

What is wrong exactly?

1. the flag `--logtostderr` exposes sensitive information to the process' stderr in a way that does not provide redactability. Consequently, plugging that output to the `kubectl log` command can defeat IT policies about confidentiality. 

   This is a serious issue that multiple customers have complained about in the 19.x cycle and we solved in 20.1/20.2 by creating product features and generally documenting that stderr logging should not be used, but somehow this operator project did not get the memo.

2. the flag `--logtostderr`  creates output that cannot be reliably parsed. Presumably, customers who integrate with orchestration wish the ability to process logging data automatically. The output produced by crdb on its stderr stream fails at this. Even though it looks like something parseable on the surface, it is a documented fact that *under stress and in problem situations, the log entries become malformed*. This is not fixable via stderr logging, but is fixable via file or network logging.  

   Right now the default config used by the k8s operator code is mistakenly giving users the impression they can rely on the logging output by `kubectl log` and automate log monitoring etc based on that data. This is a lie to our users/customers.

3. you and others who are working on the operator project are working under the mistaken impression that "logging" is somehow distinct from "database state". I see this for example in your suggestion that _"if we store log files we'd need ephemeral storage"_ or the question _"how would the operator look at the logs without execing into the node"_. That is something we need to work on culturally. 


   In a nutshell, crdb logging data has the same quality as general stored SQL data.

   We don't think about "how would the operator access the SQL data without execing into the node", right? To access SQL data, we open a SQL client. 

   The same applies to crdb logs. To access the log data, we use the log retrieval API (over HTTP).

   Generally, the log files should be treated with the same level of care for persistence *and confidentiality* as the data files: they are absolutely crucial to preserve durably, just like the crdb store data directories, and that's specificially the reason why crdb choses a `logs` sub-directory of the store directory by default. If the customer cares about at-rest confidentiality, the log files should be encrypted together with the data files.

   Right now, we are lucky that the flag `--logtostderr` merely *copies* log data to stderr but still preserves crdb's default behavior, which is to also create discrete files in the `logs` directory. The [statefulset code](https://github.com/cockroachdb/cockroach-operator/blob/master/pkg/resource/statefulset.go#L340), either intendedly or not, was not implemented to explicitly disable writing logging data to files in the store directory, and that's a good thing!

4. starting in v21.2, the `--logtostderr` flag will be removed. So the current startup code will break.

5. the best practice with distributed systems with crdb is to aggregate the logging output from all the nodes into a network logging collector. Arguably, it is the k8s operator's role to spawn a logging collector, such as Fluentd, alongside the crdb nodes inside the k8s cluster and "plug" the services together by making the logging config inside crdb point to the logging collector. This way the crdb nodes would send their data reliably, in a parseable format, to Fluentd, where it can be stored reliably and confidentially.

   This organization for log collection is what both our on-prem customers and our CC teams have requested and we've built in 21.1. It is *legitimately the best practice* for SREs and our k8s operator should definitely exemplify this best practice.

   I am surprised this hasn't been discussed yet.

So these are the problems and now let's review the suitable actions:

| Problem | Current in v20.x / v21.1 | Future severity | Action to take |
|--|--|--|--|
| (1) confidentiality leaks | yes | high (and likely to blow up when customers realize we're not delivering the level of redactability via k8s that they're expecting from our 20.2/21.1 docs) | don't use stderr logging, keep the redactable logging streams either on disk or use network logging |
| (2) broken logging format | yes | high because this will explode in our face when a customer notices that the log monitoring they've implemented was unable to catch the symptoms of a severe outage. This will persist in v21.2 because this is unfixable on our side (bug in the Go runtime system) | don't use stderr logging, use either files using the `crdb-v2` format or JSON with a network collector |
| (3) mistaken understanding about the nature of logs | yes | moderate but our Tech Support and L2 engineers will dislike the lack of log separation when they will be asked to troubleshoot k8s deployments | don't use stderr logging, and acknowledge/document log access over HTTP |
| (4) flag is deprecated | no | high when the k8s operator simply stops working with v21.2 | don't use the `--logtostderr` flag |
| (5) lack of logging cohesion | yes | moderate, will become higher when customers deploy more complex setups, including multi-region setups | don't use stderr logging, and set up an initiative to extend the k8s operator to deploy network log collectors alongside crdb nodes |

Let me know if you have further questions.


@udnay: Ok, so `--logtostderr` should be removed, @piyush-singh will provide the logging setting that are backwards compatible and provide redaction. This will use CRDBs best practices for log redaction and be backwards compatible with CRDB versions that the operator needs to support.

My strong feeing is that we need to use the premise of least surprise and that a new user should be able to see the application logs within a pod using `k logs`. As a feature enhancement we will add configurability to the CR API that will let a user fully configure their logging. For GA this is too late to add.

@knz: Some minimum amount of safe and useful logging can be provided in 21.1 by copying the events from the OPS logging channel to stderr.@piyush-singh: Okay, so I took some time to explore the solution space here and I have a few updates.

For version 21.1 and beyond, we can capture the appropriate log messages using the following log config on startup:

`cockroach start --log='sinks: {stderr: {channels: OPS, redact: true}}' <other flags>`

This will only pass redacted messages from the OPS channel to the stderr sink as Raphael pointed out above. 

For versions 20.1 and 20.2 that we are supporting on the Operator the story is a bit more complicated. I spoke with Raphael, Josh on our SRE team for Cloud, and Daniel from the SE team about expectations for K8s logs. On CockroachCloud, we actually do not log anything out to stderr (outside of the initial logs emitted on node startup) in K8s and instead the SREs exec into pods to check logs. Josh noted that this ran against his expectations initially but this has become standard practice and the team is now used to doing this. Of course, this still runs the risk that if the node is crashing, you won't be able to access logs that are persisted. Josh noted that having the full unredacted logs available _somewhere_ is important as an operator though. 

Daniel from our field team had a similar concern. When he has stood up K8s instances for customers, he usually sets up a sidecar pod to separately persist logs or streams them out to a log collection service. This lines up with our long term vision, but the complexity of setting up a sidecar for logging is too much for the GA timeline.

Raphael suggested that we may be able to backport some log redaction functionality into the 20.x releases. @knz would you please share how much of an effort you would expect that to be? I'm very hesitant to go with this route for the GA release for the operator in ~2 weeks.

In sum, we have 3 options here:
1. Keep the 20.x releases as they are with all logs going to stderr. There is a security risk inherent in doing this as those logs will contain sensitive information like PKs from the database (and much more).
2. Try to modify stderr logging behavior for 20.x release via backport.
3. Remove stderr logging, keeping only what is printed out on node startup. End users will need to exec into pods or use the HTTP logging endpoints to access logs.

Given our time constraints and that the SRE team has been using option 3 in prod in CockroachCloud, I would lean towards option 3. Versions 20.x will only be supported for another year, and depending on our TDB support policy for the operator, we may even sunset those versions earlier. After that, all versions will be able to support the above setting available from v21.1 and beyond.@knz: > @knz would you please share how much of an effort you would expect that to be? I'm very hesitant to go with this route for the GA release for the operator in ~2 weeks.

The change I have in mind is specifically to truncate log entries to just 1 lines on stderr when a flag is specified, and do pre-redaction. This would be compatible with 20.1 and 20.2, although the pre-redaction would not work on 20.1 (redactability was a 20.2 feature) and so we wouldn't be solving all the problems that way.@chrislovecnm: @piyush-singh what are we doing for the release?  This is a long issue, and sorry if it is buried in the issue somewhere@piyush-singh: Because we have 1 week until the release, I don't think we should pick up this work now. Keith pointed out that our Helm Chart and statefulset configs also have the `--logtostderr` flag set - I don't think we should try to solve this issue in the next week ahead of the release if it has been around for multiple years. I went ahead and filed #622 as a follow on enhancement after the release is out the door. I can also work with Raphael to address the v20.1 and v20.2 issues which should cover most of our bases.@PiotrKozimor: Just to give another perspective - on GKE with [Cloud Logging](https://cloud.google.com/logging) integration all logs from CRDB appear as errors.
![image](https://user-images.githubusercontent.com/37144818/138851947-1f4a521a-800a-4aa9-9c5e-11bdc807af0b.png)
@pdf: Looks like this should be closed by #827 for supported releases?@prafull01: Done with #827 

Solution patch:
diff --git a/CHANGELOG.md b/CHANGELOG.md
index c6749e3..c28978c 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -7,6 +7,10 @@ to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
 # [Unreleased](https://github.com/cockroachdb/cockroach-operator/compare/v2.5.3...master)
 
+## Added
+
+* Custom logging configuration can be used through the configmap when `spec.logConfigMap` is provided.
+
 # [v2.5.3](https://github.com/cockroachdb/cockroach-operator/compare/v2.5.2...v2.5.3)
 
 ## Fixed
diff --git a/apis/v1alpha1/cluster_types.go b/apis/v1alpha1/cluster_types.go
index 518ed2b..0d4812c 100644
--- a/apis/v1alpha1/cluster_types.go
+++ b/apis/v1alpha1/cluster_types.go
@@ -130,6 +130,13 @@ type CrdbClusterSpec struct {
 	// +operator-sdk:csv:customresourcedefinitions:type=spec,displayName="Cockroach Database Ingress"
 	// +optional
 	Ingress *IngressConfig `json:"ingress,omitempty"`
+	// (Optional) LogConfigMap define the config map which contains log configuration used to send the logs through the
+	// proper channels in the cockroachdb. Logging configuration is available for cockroach version v21.1.0 onwards.
+	// The logging configuration is taken in format of yaml file, you can check the logging configuration here (https://www.cockroachlabs.com/docs/stable/configure-logs.html#default-logging-configuration)
+	// The default logging for cockroach version v20.x or less is stderr, logging API is ignored for older versions.
+	// +operator-sdk:csv:customresourcedefinitions:type=spec,displayName="Cockroach Database Logging configuration config map"
+	// +optional
+	LogConfigMap string `json:"logConfigMap,omitempty"`
 }
 
 // +k8s:openapi-gen=true
diff --git a/apis/v1alpha1/webhook.go b/apis/v1alpha1/webhook.go
index b5628ae..dc4f8fc 100644
--- a/apis/v1alpha1/webhook.go
+++ b/apis/v1alpha1/webhook.go
@@ -180,4 +180,4 @@ func (r *CrdbCluster) ValidateVolumeMode() error {
 			"you have provided unsupported pvc.volumeMode, currently only Filesystem is supported.")
 	}
 	return nil
-}
\ No newline at end of file
+}
diff --git a/config/crd/bases/crdb.cockroachlabs.com_crdbclusters.yaml b/config/crd/bases/crdb.cockroachlabs.com_crdbclusters.yaml
index 4797b37..b0068fc 100644
--- a/config/crd/bases/crdb.cockroachlabs.com_crdbclusters.yaml
+++ b/config/crd/bases/crdb.cockroachlabs.com_crdbclusters.yaml
@@ -973,6 +973,15 @@ spec:
                     - host
                     type: object
                 type: object
+              logConfigMap:
+                description: (Optional) LogConfigMap define the config map which contains
+                  log configuration used to send the logs through the proper channels
+                  in the cockroachdb. Logging configuration is available for cockroach
+                  version v21.1.0 onwards. The logging configuration is taken in format
+                  of yaml file, you can check the logging configuration here (https://www.cockroachlabs.com/docs/stable/configure-logs.html#default-logging-configuration)
+                  The default logging for cockroach version v20.x or less is stderr,
+                  logging API is ignored for older versions.
+                type: string
               maxSQLMemory:
                 description: '(Optional) The maximum in-memory storage capacity available
                   to store temporary data for SQL queries (`--max-sql-memory` parameter)
diff --git a/config/rbac/role.yaml b/config/rbac/role.yaml
index 7b7089a..f3ecce6 100644
--- a/config/rbac/role.yaml
+++ b/config/rbac/role.yaml
@@ -128,6 +128,8 @@ rules:
   - configmaps
   verbs:
   - get
+  - list
+  - watch
 - apiGroups:
   - ""
   resources:
diff --git a/e2e/versionchecker/BUILD.bazel b/e2e/versionchecker/BUILD.bazel
index b20912b..a86f2af 100644
--- a/e2e/versionchecker/BUILD.bazel
+++ b/e2e/versionchecker/BUILD.bazel
@@ -5,6 +5,7 @@ go_test(
     size = "enormous",
     srcs = ["versionchecker_test.go"],
     deps = [
+        "//apis/v1alpha1:go_default_library",
         "//e2e:go_default_library",
         "//pkg/controller:go_default_library",
         "//pkg/testutil:go_default_library",
@@ -13,6 +14,7 @@ go_test(
         "@com_github_stretchr_testify//require:go_default_library",
         "@io_k8s_api//core/v1:go_default_library",
         "@io_k8s_apimachinery//pkg/api/resource:go_default_library",
+        "@io_k8s_apimachinery//pkg/apis/meta/v1:go_default_library",
         "@org_uber_go_zap//zaptest:go_default_library",
     ],
 )
diff --git a/hack/bin/deps.bzl b/hack/bin/deps.bzl
index 4255bb8..dc0e5f3 100644
--- a/hack/bin/deps.bzl
+++ b/hack/bin/deps.bzl
@@ -442,13 +442,13 @@ filegroup(
 def install_crdb():
     http_archive(
        name = "crdb_darwin", # todo fix or remove
-       sha256 = "bbe3a03c661555e8b083856c56c8a3b459f83064d1d552ed3467cbfb66e76db7",
-       urls = ["https://binaries.cockroachdb.com/cockroach-v20.2.5.darwin-10.9-amd64.tgz"],
+       sha256 = "bbbd0a75f81d3df4acd139fdc7f0961480161454db24f25263c9276c3959db54",
+       urls = ["https://binaries.cockroachdb.com/cockroach-v21.2.0.darwin-10.9-amd64.tgz"],
        build_file_content = """
 filegroup(
     name = "file",
     srcs = [
-        "cockroach-v20.2.5.darwin-10.9-amd64/cockroach",
+        "cockroach-v21.2.0.darwin-10.9-amd64/cockroach",
     ],
     visibility = ["//visibility:public"],
 )
@@ -457,13 +457,13 @@ filegroup(
 
     http_archive(
         name = "crdb_linux",
-        sha256 = "57f4b00c736d8511328d6f33997a3a66cb4ec7142cb126d872dade399a0922e6",
-        urls = ["https://binaries.cockroachdb.com/cockroach-v20.2.5.linux-amd64.tgz"],
+        sha256 = "c9fda447b9db98ade4444f5855ceb6ffe94549a20bd7cad8fdf70c398add8c02",
+        urls = ["https://binaries.cockroachdb.com/cockroach-v21.2.0.linux-amd64.tgz"],
         build_file_content = """
 filegroup(
     name = "file",
     srcs = [
-        "cockroach-v20.2.5.linux-amd64/cockroach",
+        "cockroach-v21.2.0.linux-amd64/cockroach",
     ],
     visibility = ["//visibility:public"],
 )
diff --git a/install/crds.yaml b/install/crds.yaml
index a8b1d3f..e572c24 100644
--- a/install/crds.yaml
+++ b/install/crds.yaml
@@ -971,6 +971,15 @@ spec:
                     - host
                     type: object
                 type: object
+              logConfigMap:
+                description: (Optional) LogConfigMap define the config map which contains
+                  log configuration used to send the logs through the proper channels
+                  in the cockroachdb. Logging configuration is available for cockroach
+                  version v21.1.0 onwards. The logging configuration is taken in format
+                  of yaml file, you can check the logging configuration here (https://www.cockroachlabs.com/docs/stable/configure-logs.html#default-logging-configuration)
+                  The default logging for cockroach version v20.x or less is stderr,
+                  logging API is ignored for older versions.
+                type: string
               maxSQLMemory:
                 description: '(Optional) The maximum in-memory storage capacity available
                   to store temporary data for SQL queries (`--max-sql-memory` parameter)
diff --git a/install/operator.yaml b/install/operator.yaml
index de47ec6..4c6fae8 100644
--- a/install/operator.yaml
+++ b/install/operator.yaml
@@ -141,6 +141,8 @@ rules:
   - configmaps
   verbs:
   - get
+  - list
+  - watch
 - apiGroups:
   - ""
   resources:
diff --git a/pkg/actor/cluster_restart.go b/pkg/actor/cluster_restart.go
index 7c34081..63309eb 100644
--- a/pkg/actor/cluster_restart.go
+++ b/pkg/actor/cluster_restart.go
@@ -117,6 +117,7 @@ func (r *clusterRestart) Act(ctx context.Context, cluster *resource.Cluster, log
 		return err
 	}
 	refreshedCluster := resource.NewCluster(cr)
+	refreshedCluster.Fetcher = fetcher
 	//delete annotation
 	refreshedCluster.DeleteRestartTypeAnnotation()
 	//TODO  use patch for annotations
diff --git a/pkg/actor/generate_cert.go b/pkg/actor/generate_cert.go
index 1926c08..138cbfb 100644
--- a/pkg/actor/generate_cert.go
+++ b/pkg/actor/generate_cert.go
@@ -124,6 +124,7 @@ func (rc *generateCert) Act(ctx context.Context, cluster *resource.Cluster, log
 			return errors.Wrap(err, msg)
 		}
 		refreshedCluster := resource.NewCluster(newcr)
+		refreshedCluster.Fetcher = fetcher
 		refreshedCluster.SetAnnotationCertExpiration(*expirationDatePtr)
 		if restartRequired {
 			refreshedCluster.SetRestartTypeAnnotation(api.ClusterRestartType(api.RollingRestart).String())
@@ -154,6 +155,7 @@ func (rc *generateCert) Act(ctx context.Context, cluster *resource.Cluster, log
 			return errors.Wrap(err, msg)
 		}
 		refreshedCluster := resource.NewCluster(newcr)
+		refreshedCluster.Fetcher = fetcher
 		refreshedCluster.SetTrue(api.CertificateGenerated)
 		if refreshedCluster.IsSQLIngressEnabled() {
 			refreshedCluster.SetSQLHost(cluster.Spec().Ingress.SQL.Host)
diff --git a/pkg/actor/initialize.go b/pkg/actor/initialize.go
index 237a5a2..7ac518f 100644
--- a/pkg/actor/initialize.go
+++ b/pkg/actor/initialize.go
@@ -131,6 +131,7 @@ func (init initialize) Act(ctx context.Context, cluster *resource.Cluster, log l
 
 		// Set the CrdbInitialized condition in the cluster status to true
 		refreshedCluster := resource.NewCluster(newcr)
+		refreshedCluster.Fetcher = fetcher
 		refreshedCluster.SetTrue(api.CrdbInitializedCondition)
 
 		// Actually attempt to update the CrdbClusterStatus object. If the update runs into a conflict for any reason
diff --git a/pkg/actor/validate_version.go b/pkg/actor/validate_version.go
index b2da66f..ced485a 100644
--- a/pkg/actor/validate_version.go
+++ b/pkg/actor/validate_version.go
@@ -21,6 +21,7 @@ import (
 	"context"
 	"fmt"
 	"io"
+	"os/exec"
 	"strings"
 	"time"
 
@@ -59,6 +60,30 @@ func (v *versionChecker) GetActionType() api.ActionType {
 }
 
 func (v *versionChecker) Act(ctx context.Context, cluster *resource.Cluster, log logr.Logger) error {
+	log.V(DEBUGLEVEL).Info("starting to check the logging config provided")
+	//we refresh the resource to make sure we use the latest version
+	fetcher := resource.NewKubeFetcher(ctx, cluster.Namespace(), v.client)
+
+	if cluster.IsLoggingAPIEnabled() {
+		if logConfig, err := cluster.LoggingConfiguration(fetcher); err == nil {
+			log.V(DEBUGLEVEL).Info(fmt.Sprintf("Log configuration for the cockroach cluster: %s", logConfig))
+			var stderr bytes.Buffer
+			cmd := exec.Command("bash", "-c", fmt.Sprintf("cockroach debug check-log-config --log=%s", logConfig))
+			cmd.Stderr = &stderr
+			cErr := cmd.Run()
+			if cErr != nil || stderr.String() != "" {
+				log.Error(cErr, "The cockroachdb logging API is set to value that is not supported by the operator, See the default logging configuration here (https://www.cockroachlabs.com/docs/stable/configure-logs.html#default-logging-configuration) ")
+				return errors.New(stderr.String())
+			} else {
+				log.V(DEBUGLEVEL).Info("Validated the logging config")
+			}
+		} else {
+			vErr := ValidationError{Err: err}
+			log.Error(vErr, "The cockroachdb logging API value is set to a value that is not supported by the operator")
+			return err
+		}
+	}
+
 	log.V(DEBUGLEVEL).Info("starting to check the crdb version of the container provided")
 
 	r := resource.NewManagedKubeResource(ctx, v.client, cluster, kube.AnnotatingPersister)
@@ -272,6 +297,7 @@ func (v *versionChecker) Act(ctx context.Context, cluster *resource.Cluster, log
 		}
 
 		refreshedCluster := resource.NewCluster(cr)
+		refreshedCluster.Fetcher = fetcher
 		refreshedCluster.SetClusterVersion(calVersion)
 		refreshedCluster.SetAnnotationVersion(calVersion)
 		refreshedCluster.SetCrdbContainerImage(containerImage)
@@ -312,6 +338,7 @@ func (v *versionChecker) completeVersionChecker(
 	}
 
 	refreshedCluster := resource.NewCluster(cr)
+	refreshedCluster.Fetcher = fetcher
 	// save the status of the cluster
 	refreshedCluster.SetTrue(api.CrdbVersionChecked)
 	refreshedCluster.SetClusterVersion(version)
diff --git a/pkg/controller/cluster_controller.go b/pkg/controller/cluster_controller.go
index 55d4a54..f073546 100644
--- a/pkg/controller/cluster_controller.go
+++ b/pkg/controller/cluster_controller.go
@@ -60,7 +60,7 @@ type ClusterReconciler struct {
 // +kubebuilder:rbac:groups=core,resources=services,verbs=get;list;watch;create;update;patch;delete
 // +kubebuilder:rbac:groups=core,resources=services/status,verbs=get;update;patch
 // +kubebuilder:rbac:groups=core,resources=services/finalizers,verbs=get;list;watch
-// +kubebuilder:rbac:groups=core,resources=configmaps,verbs=get
+// +kubebuilder:rbac:groups=core,resources=configmaps,verbs=get;list;watch
 // +kubebuilder:rbac:groups=core,resources=configmaps/status,verbs=get
 // +kubebuilder:rbac:groups=core,resources=secrets,verbs=get;list;watch;create;update;patch
 // +kubebuilder:rbac:groups=core,resources=pods,verbs=get;list
@@ -114,6 +114,7 @@ func (r *ClusterReconciler) Reconcile(ctx context.Context, req reconcile.Request
 	}
 
 	cluster := resource.NewCluster(cr)
+	cluster.Fetcher = fetcher
 	// on first run we need to save the status and exit to pass Openshift CI
 	// we added a state called Starting for field ClusterStatus to accomplish this
 	if cluster.Status().ClusterStatus == "" {
diff --git a/pkg/resource/BUILD.bazel b/pkg/resource/BUILD.bazel
index 77747d3..0c10a1f 100644
--- a/pkg/resource/BUILD.bazel
+++ b/pkg/resource/BUILD.bazel
@@ -32,6 +32,7 @@ go_library(
         "@com_github_cockroachdb_errors//:go_default_library",
         "@com_github_go_logr_logr//:go_default_library",
         "@com_github_gosimple_slug//:go_default_library",
+        "@com_github_masterminds_semver_v3//:go_default_library",
         "@io_k8s_api//admissionregistration/v1:go_default_library",
         "@io_k8s_api//apps/v1:go_default_library",
         "@io_k8s_api//batch/v1:go_default_library",
diff --git a/pkg/resource/cluster.go b/pkg/resource/cluster.go
index 198e164..f190dce 100644
--- a/pkg/resource/cluster.go
+++ b/pkg/resource/cluster.go
@@ -22,6 +22,7 @@ import (
 	"strings"
 	"time"
 
+	"github.com/Masterminds/semver/v3"
 	api "github.com/cockroachdb/cockroach-operator/apis/v1alpha1"
 	"github.com/cockroachdb/cockroach-operator/pkg/clusterstatus"
 	"github.com/cockroachdb/cockroach-operator/pkg/condition"
@@ -319,6 +320,47 @@ func (cluster Cluster) IsFresh(fetcher Fetcher) (bool, error) {
 	return cluster.cr.ResourceVersion == actual.ResourceVersion, nil
 }
 
+func (cluster Cluster) LoggingConfiguration(fetcher Fetcher) (string, error) {
+	if cluster.Spec().LogConfigMap != "" {
+		cm := &corev1.ConfigMap{
+			ObjectMeta: metav1.ObjectMeta{
+				Name: cluster.Spec().LogConfigMap,
+			},
+		}
+		err := fetcher.Fetch(cm)
+		if err != nil {
+			return "", err
+		}
+
+		if val, ok := cm.Data["logging.yaml"]; ok {
+			return `"` + val + `"`, nil
+		}
+	}
+
+	return "\"{sinks: {stderr: {channels: [OPS, HEALTH], redact: true}}}\"", nil
+}
+
+func (cluster Cluster) IsLoggingAPIEnabled() bool {
+	var version string
+	if cluster.Spec().CockroachDBVersion != "" {
+		version = cluster.Spec().CockroachDBVersion
+	} else if cluster.Spec().Image != nil && cluster.Spec().Image.Name != "" {
+		version = strings.Split(cluster.Spec().Image.Name, ":")[1]
+	} else {
+		return false
+	}
+
+	// No need to handle the error as the version provided is constant
+	c, _ := semver.NewConstraint(">= v21.1")
+
+	v, err := semver.NewVersion(version)
+	if err != nil {
+		return false
+	}
+
+	return c.Check(v)
+}
+
 // TODO add error handling to ensure that env variables are set correctly and
 // that we have a min number of them
 
diff --git a/pkg/resource/statefulset.go b/pkg/resource/statefulset.go
index 19aad2a..8d13d65 100644
--- a/pkg/resource/statefulset.go
+++ b/pkg/resource/statefulset.go
@@ -360,13 +360,19 @@ func (b StatefulSetBuilder) dbArgs() []string {
 		"--join=" + b.joinStr(),
 		fmt.Sprintf("--advertise-host=$(POD_NAME).%s.%s",
 			b.Cluster.DiscoveryServiceName(), b.Cluster.Namespace()),
-		"--logtostderr=INFO",
 		b.Cluster.SecureMode(),
 		"--http-port=" + fmt.Sprint(*b.Spec().HTTPPort),
 		"--sql-addr=:" + fmt.Sprint(*b.Spec().SQLPort),
 		"--listen-addr=:" + fmt.Sprint(*b.Spec().GRPCPort),
 	}
 
+	if b.Cluster.IsLoggingAPIEnabled() {
+		logConfig, _ := b.Cluster.LoggingConfiguration(b.Cluster.Fetcher)
+		aa = append(aa, fmt.Sprintf("--log=%s", logConfig))
+	} else {
+		aa = append(aa, "--logtostderr=INFO")
+	}
+
 	if b.Spec().Cache != "" {
 		aa = append(aa, "--cache="+b.Spec().Cache)
 	} else {



Test patch:
diff --git a/apis/v1alpha1/webhook_test.go b/apis/v1alpha1/webhook_test.go
index f0f197b..0f46f25 100644
--- a/apis/v1alpha1/webhook_test.go
+++ b/apis/v1alpha1/webhook_test.go
@@ -102,8 +102,7 @@ func TestCreateCrdbCluster(t *testing.T) {
 				Image: &PodImage{Name: "testImage"},
 				DataStore: Volume{
 					VolumeClaim: &VolumeClaim{
-						PersistentVolumeClaimSpec: v1.PersistentVolumeClaimSpec{
-						},
+						PersistentVolumeClaimSpec: v1.PersistentVolumeClaimSpec{},
 					},
 				},
 			}},
diff --git a/e2e/create/create_test.go b/e2e/create/create_test.go
index 60eb3de..18cb528 100644
--- a/e2e/create/create_test.go
+++ b/e2e/create/create_test.go
@@ -17,6 +17,7 @@ limitations under the License.
 package e2e
 
 import (
+	"encoding/json"
 	"os"
 	"testing"
 
@@ -54,8 +55,14 @@ func TestCreateInsecureCluster(t *testing.T) {
 	sb := testenv.NewDiffingSandbox(t, env)
 	sb.StartManager(t, controller.InitClusterReconcilerWithLogger(testLog))
 
+	// Create cluster with different logging config than the default one.
+	logJson := []byte(`{"sinks": {"file-groups": {"dev": {"channels": "DEV", "filter": "WARNING"}}}}`)
+	logConfig := make(map[string]interface{})
+	require.NoError(t, json.Unmarshal(logJson, &logConfig))
+	testutil.RequireLoggingConfigMap(t, sb, "logging-configmap", string(logJson))
+
 	builder := testutil.NewBuilder("crdb").WithNodeCount(3).
-		WithImage(e2e.MajorVersion).
+		WithImage(e2e.MajorVersion).WithClusterLogging("logging-configmap").
 		WithPVDataStore("1Gi", "standard" /* default storage class in KIND */)
 
 	steps := testutil.Steps{
diff --git a/e2e/versionchecker/versionchecker_test.go b/e2e/versionchecker/versionchecker_test.go
index e5e0b5b..09a4a50 100644
--- a/e2e/versionchecker/versionchecker_test.go
+++ b/e2e/versionchecker/versionchecker_test.go
@@ -17,12 +17,15 @@ limitations under the License.
 package versionchecker_test
 
 import (
+	"encoding/json"
 	"testing"
 	"time"
 
 	corev1 "k8s.io/api/core/v1"
 	apiresource "k8s.io/apimachinery/pkg/api/resource"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 
+	crdbv1alpha1 "github.com/cockroachdb/cockroach-operator/apis/v1alpha1"
 	"github.com/cockroachdb/cockroach-operator/e2e"
 	"github.com/cockroachdb/cockroach-operator/pkg/controller"
 	"github.com/cockroachdb/cockroach-operator/pkg/testutil"
@@ -70,3 +73,47 @@ func TestVersionCheckerJobPodPending(t *testing.T) {
 	}
 	steps.Run(t)
 }
+
+func TestLoggingAPIValidCheck(t *testing.T) {
+	if testing.Short() {
+		t.Skip("skipping test in short mode.")
+	}
+	testLog := zapr.NewLogger(zaptest.NewLogger(t))
+
+	e := testenv.CreateActiveEnvForTest()
+	env := e.Start()
+	defer e.Stop()
+
+	sb := testenv.NewDiffingSandbox(t, env)
+	sb.StartManager(t, controller.InitClusterReconcilerWithLogger(testLog))
+
+	logJson := []byte(`{"sinkss": "invalidvalue"}`)
+	logConfig := make(map[string]interface{})
+	require.NoError(t, json.Unmarshal(logJson, &logConfig))
+	testutil.RequireLoggingConfigMap(t, sb, "logging-configmap", string(logJson))
+
+	builder := testutil.NewBuilder("crdb").Namespaced(sb.Namespace).WithNodeCount(3).WithTLS().
+		WithImage("cockroachdb/cockroach:v21.1.0").
+		WithPVDataStore("32Mi", "standard" /* default storage class in KIND */).
+		WithClusterLogging("logging-configmap")
+
+	steps := testutil.Steps{
+		{
+			Name: "validate the logging API input",
+			Test: func(t *testing.T) {
+				require.NoError(t, sb.Create(builder.Cr()))
+				crdbCluster := &crdbv1alpha1.CrdbCluster{
+					ObjectMeta: metav1.ObjectMeta{
+						Name:      builder.Cr().Name,
+						Namespace: builder.Cr().Namespace,
+					},
+				}
+				// Added sleep to reconcile the CR.
+				time.Sleep(2 * time.Second)
+				require.NoError(t, sb.Get(crdbCluster))
+				require.Equal(t, crdbCluster.Status.ClusterStatus, "Failed")
+			},
+		},
+	}
+	steps.Run(t)
+}
diff --git a/pkg/resource/testdata/TestStatefulSetBuilder/default_insecure_statefulset.golden b/pkg/resource/testdata/TestStatefulSetBuilder/default_insecure_statefulset.golden
index d45d663..df43e1e 100755
--- a/pkg/resource/testdata/TestStatefulSetBuilder/default_insecure_statefulset.golden
+++ b/pkg/resource/testdata/TestStatefulSetBuilder/default_insecure_statefulset.golden
@@ -28,7 +28,7 @@ spec:
       - command:
         - /bin/bash
         - -ecx
-        - exec /cockroach/cockroach.sh start --join=test-cluster-0.test-cluster.test-ns:26258 --advertise-host=$(POD_NAME).test-cluster.test-ns --logtostderr=INFO --insecure --http-port=8080 --sql-addr=:26257 --listen-addr=:26258 --cache $(expr $MEMORY_LIMIT_MIB / 4)MiB --max-sql-memory $(expr $MEMORY_LIMIT_MIB / 4)MiB
+        - "exec /cockroach/cockroach.sh start --join=test-cluster-0.test-cluster.test-ns:26258 --advertise-host=$(POD_NAME).test-cluster.test-ns --insecure --http-port=8080 --sql-addr=:26257 --listen-addr=:26258 --log=\"{sinks: {stderr: {channels: [OPS, HEALTH], redact: true}}}\" --cache $(expr $MEMORY_LIMIT_MIB / 4)MiB --max-sql-memory $(expr $MEMORY_LIMIT_MIB / 4)MiB"
         env:
         - name: COCKROACH_CHANNEL
           value: kubernetes-operator-gke
@@ -46,7 +46,7 @@ spec:
             resourceFieldRef:
               divisor: 1Mi
               resource: limits.memory
-        image: cockroachdb/cockroach:v20.2.7
+        image: cockroachdb/cockroach:v21.1.0
         imagePullPolicy: IfNotPresent
         lifecycle:
           preStop:
diff --git a/pkg/resource/testdata/TestStatefulSetBuilder/default_insecure_statefulset_in.yaml b/pkg/resource/testdata/TestStatefulSetBuilder/default_insecure_statefulset_in.yaml
index 1f5d1f9..3370b9a 100644
--- a/pkg/resource/testdata/TestStatefulSetBuilder/default_insecure_statefulset_in.yaml
+++ b/pkg/resource/testdata/TestStatefulSetBuilder/default_insecure_statefulset_in.yaml
@@ -31,7 +31,7 @@ spec:
   grpcPort: 26258
   httpPort: 8080
   image:
-    name: cockroachdb/cockroach:v20.2.7
+    name: cockroachdb/cockroach:v21.1.0
   nodes: 1
   topology:
     zones:
diff --git a/pkg/resource/testdata/TestStatefulSetBuilder/default_secure.golden b/pkg/resource/testdata/TestStatefulSetBuilder/default_secure.golden
index 59264b3..d489d03 100755
--- a/pkg/resource/testdata/TestStatefulSetBuilder/default_secure.golden
+++ b/pkg/resource/testdata/TestStatefulSetBuilder/default_secure.golden
@@ -30,7 +30,7 @@ spec:
       - command:
         - /bin/bash
         - -ecx
-        - exec /cockroach/cockroach.sh start --join=test-cluster-0.test-cluster.test-ns:26258 --advertise-host=$(POD_NAME).test-cluster.test-ns --logtostderr=INFO --certs-dir=/cockroach/cockroach-certs/ --http-port=8080 --sql-addr=:26257 --listen-addr=:26258 --cache $(expr $MEMORY_LIMIT_MIB / 4)MiB --max-sql-memory $(expr $MEMORY_LIMIT_MIB / 4)MiB
+        - "exec /cockroach/cockroach.sh start --join=test-cluster-0.test-cluster.test-ns:26258 --advertise-host=$(POD_NAME).test-cluster.test-ns --certs-dir=/cockroach/cockroach-certs/ --http-port=8080 --sql-addr=:26257 --listen-addr=:26258 --log=\"{sinks: {stderr: {channels: [OPS, HEALTH], redact: true}}}\" --cache $(expr $MEMORY_LIMIT_MIB / 4)MiB --max-sql-memory $(expr $MEMORY_LIMIT_MIB / 4)MiB"
         env:
         - name: COCKROACH_CHANNEL
           value: kubernetes-operator-gke
@@ -48,7 +48,7 @@ spec:
             resourceFieldRef:
               divisor: 1Mi
               resource: limits.memory
-        image: cockroachdb/cockroach:v20.2.7
+        image: cockroachdb/cockroach:v21.1.0
         imagePullPolicy: IfNotPresent
         lifecycle:
           preStop:
@@ -87,7 +87,7 @@ spec:
         - /bin/sh
         - -c
         - '>- cp -p /cockroach/cockroach-certs-prestage/..data/* /cockroach/cockroach-certs/ && chmod 700 /cockroach/cockroach-certs/*.key && chown 1000581000:1000581000 /cockroach/cockroach-certs/*.key'
-        image: cockroachdb/cockroach:v20.2.7
+        image: cockroachdb/cockroach:v21.1.0
         imagePullPolicy: IfNotPresent
         name: db-init
         resources: {}
diff --git a/pkg/resource/testdata/TestStatefulSetBuilder/default_secure_in.yaml b/pkg/resource/testdata/TestStatefulSetBuilder/default_secure_in.yaml
index 55b43b0..39a780c 100644
--- a/pkg/resource/testdata/TestStatefulSetBuilder/default_secure_in.yaml
+++ b/pkg/resource/testdata/TestStatefulSetBuilder/default_secure_in.yaml
@@ -31,7 +31,7 @@ spec:
   grpcPort: 26258
   httpPort: 8080
   image:
-    name: cockroachdb/cockroach:v20.2.7
+    name: cockroachdb/cockroach:v21.1.0
   nodes: 1
   tlsEnabled: true
   topology:
diff --git a/pkg/resource/testdata/TestStatefulSetBuilder/insecure_statefulset_cli_args.golden b/pkg/resource/testdata/TestStatefulSetBuilder/insecure_statefulset_cli_args.golden
index 7594638..75d0d1b 100755
--- a/pkg/resource/testdata/TestStatefulSetBuilder/insecure_statefulset_cli_args.golden
+++ b/pkg/resource/testdata/TestStatefulSetBuilder/insecure_statefulset_cli_args.golden
@@ -47,7 +47,7 @@ spec:
       - command:
         - /bin/bash
         - -ecx
-        - exec /cockroach/cockroach.sh start --join=test-cluster-0.test-cluster.test-ns:26258 --advertise-host=$(POD_NAME).test-cluster.test-ns --logtostderr=INFO --insecure --http-port=8080 --sql-addr=:26257 --listen-addr=:26258 --cache=30% --max-sql-memory=2GB --temp-dir=/tmp
+        - "exec /cockroach/cockroach.sh start --join=test-cluster-0.test-cluster.test-ns:26258 --advertise-host=$(POD_NAME).test-cluster.test-ns --insecure --http-port=8080 --sql-addr=:26257 --listen-addr=:26258 --log=\"{sinks: {stderr: {channels: [OPS, HEALTH], redact: true}}}\" --cache=30% --max-sql-memory=2GB --temp-dir=/tmp"
         env:
         - name: COCKROACH_CHANNEL
           value: kubernetes-operator-gke
@@ -65,7 +65,7 @@ spec:
             resourceFieldRef:
               divisor: 1Mi
               resource: limits.memory
-        image: cockroachdb/cockroach:v20.2.7
+        image: cockroachdb/cockroach:v21.1.0
         imagePullPolicy: IfNotPresent
         lifecycle:
           preStop:
diff --git a/pkg/resource/testdata/TestStatefulSetBuilder/insecure_statefulset_cli_args_in.yaml b/pkg/resource/testdata/TestStatefulSetBuilder/insecure_statefulset_cli_args_in.yaml
index 9185c25..42cddf2 100644
--- a/pkg/resource/testdata/TestStatefulSetBuilder/insecure_statefulset_cli_args_in.yaml
+++ b/pkg/resource/testdata/TestStatefulSetBuilder/insecure_statefulset_cli_args_in.yaml
@@ -31,7 +31,7 @@ spec:
   grpcPort: 26258
   httpPort: 8080
   image:
-    name: cockroachdb/cockroach:v20.2.7
+    name: cockroachdb/cockroach:v21.1.0
   cache: 30%
   maxSQLMemory: 2GB
   additionalArgs:
diff --git a/pkg/resource/testdata/TestStatefulSetBuilder/insecure_statefulset_with_resources.golden b/pkg/resource/testdata/TestStatefulSetBuilder/insecure_statefulset_with_resources.golden
index 3b0f402..1b81abb 100755
--- a/pkg/resource/testdata/TestStatefulSetBuilder/insecure_statefulset_with_resources.golden
+++ b/pkg/resource/testdata/TestStatefulSetBuilder/insecure_statefulset_with_resources.golden
@@ -28,7 +28,7 @@ spec:
       - command:
         - /bin/bash
         - -ecx
-        - exec /cockroach/cockroach.sh start --join=test-cluster-0.test-cluster.test-ns:26258 --advertise-host=$(POD_NAME).test-cluster.test-ns --logtostderr=INFO --insecure --http-port=8080 --sql-addr=:26257 --listen-addr=:26258 --cache $(expr $MEMORY_LIMIT_MIB / 4)MiB --max-sql-memory $(expr $MEMORY_LIMIT_MIB / 4)MiB
+        - exec /cockroach/cockroach.sh start --join=test-cluster-0.test-cluster.test-ns:26258 --advertise-host=$(POD_NAME).test-cluster.test-ns --insecure --http-port=8080 --sql-addr=:26257 --listen-addr=:26258 --logtostderr=INFO --cache $(expr $MEMORY_LIMIT_MIB / 4)MiB --max-sql-memory $(expr $MEMORY_LIMIT_MIB / 4)MiB
         env:
         - name: COCKROACH_CHANNEL
           value: kubernetes-operator-gke
diff --git a/pkg/testutil/builder.go b/pkg/testutil/builder.go
index 11c0859..d401da5 100644
--- a/pkg/testutil/builder.go
+++ b/pkg/testutil/builder.go
@@ -127,6 +127,11 @@ func (b ClusterBuilder) WithClusterAnnotations(annotations map[string]string) Cl
 	return b
 }
 
+func (b ClusterBuilder) WithClusterLogging(logConfigMap string) ClusterBuilder {
+	b.cluster.Spec.LogConfigMap = logConfigMap
+	return b
+}
+
 func (b ClusterBuilder) WithAnnotations(annotations map[string]string) ClusterBuilder {
 	b.cluster.Spec.AdditionalAnnotations = annotations
 	return b
diff --git a/pkg/testutil/require.go b/pkg/testutil/require.go
index 40b30a5..f35af6f 100644
--- a/pkg/testutil/require.go
+++ b/pkg/testutil/require.go
@@ -654,3 +654,18 @@ func RequireClusterInFailedState(t *testing.T, sb testenv.DiffingSandbox, b Clus
 
 	require.NoError(t, wErr)
 }
+
+func RequireLoggingConfigMap(t *testing.T, sb testenv.DiffingSandbox, name string, logConfig string) {
+	var loggingConfigMap = corev1.ConfigMap{
+		ObjectMeta: metav1.ObjectMeta{
+			Name:      name,
+			Namespace: sb.Namespace,
+		},
+		Data: map[string]string{
+			"logging.yaml": logConfig,
+		},
+	}
+
+	err := sb.Create(&loggingConfigMap)
+	require.NoError(t, err)
+}