Repository: rook/rook
Pull request #14305 (2024-06-04 15:14:05)

csi: add log rotation for csi rbd pod containers
1) Make the csi rbd container logs persisted in a file
   (csi plugin, csi provisioner, csi addons sidecar)

2) Use the cephcluster api specs to configure the log rotate

3) Add log rotation to rotate the log file and  
   Add a sidecar log collector container

part-of: https://github.com/rook/rook/issues/12809

Updates: 
- --logtostderr=false
- --alsologtostderr=true
- --log_file=/var/lib/cephcsi/log/rbdprov.log

```
sh-4.4# cat csi
/dev/*.log {
  rotate 3
  daily
  size 1M
  copytruncate
  start 1
}
```

When you mount a volume to a directory in a container, it effectively overrides the contents of that directory with the contents of the volume. This can cause issues if there are existing configurations or files in that directory that you want to preserve while adding new files or configurations.

<!-- Thank you for contributing to Rook! -->

<!-- STEPS TO FOLLOW:
  1. Add a description of the changes (frequently the same as the commit description)
  2. Enter the issue number next to "Resolves #" below (if there is no tracking issue resolved, **remove that section**)
  3. Review our Contributing documentation at https://rook.io/docs/rook/latest/Contributing/development-flow/
  4. Follow the steps in the checklist below, starting with the **Commit Message Formatting**.
-->

<!-- Uncomment this section with the issue number if an issue is being resolved
**Issue resolved by this Pull Request:**
Resolves #
--->

**Checklist:**

- [ ] **Commit Message Formatting**: Commit titles and messages follow guidelines in the [developer guide](https://rook.io/docs/rook/latest/Contributing/development-flow/#commit-structure).
- [ ] Reviewed the developer guide on [Submitting a Pull Request](https://rook.io/docs/rook/latest/Contributing/development-flow/#submitting-a-pull-request)
- [ ] [Pending release notes](https://github.com/rook/rook/blob/master/PendingReleaseNotes.md) updated with breaking and/or notable changes for the next minor release.
- [ ] Documentation has been updated, if necessary.
- [ ] Unit tests have been added, if necessary.
- [ ] Integration tests have been added, if necessary.


parth-gr: > I'd like to get @Madhu-1 's thought's on this as well, but it seems to me that adding these changes to all 4 major CSI deployment templates will be a lot of boilerplate to maintain over time. It seems like it might be in the interest of future devs to define this as a common Golang template that gets applied after template loading.

can we have a common shell script
Madhu-1: > I'd like to get @Madhu-1 's thought's on this as well, but it seems to me that adding these changes to all 4 major CSI deployment templates will be a lot of boilerplate to maintain over time. It seems like it might be in the interest of future devs to define this as a common Golang template that gets applied after template loading.

@BlaineEXE  sorry missed the ping. yes what you said is the right way to do it, we can define a new yaml object once and reuse in all the places which avoids application and maintenance burdens.
parth-gr: Testing:

For rbd proviosioner pod:

```
[rider@localhost examples]$ kubectl exec -it csi-rbdplugin-provisioner-5586ffc799-tg2pd -nrook-ceph -c log-collector sh
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
sh-4.4# 
sh-4.4# cd /var/
adm/      crash/    empty/    games/    kerberos/ local/    log/      nis/      preserve/ spool/    tmp/      
cache/    db/       ftp/      gopher/   lib/      lock/     mail/     opt/      run/      target/   yp/       
sh-4.4# cd /var/lib/cephcsi/logrotate-config/rbd-provisioner/    
sh-4.4# cat csi 
/var/lib/cephcsi/log/rbd-provisioner/*.log {
    hourly
    missingok
    maxsize 1M
    rotate 7
    compress
    copytruncate
    notifempty
}
sh-4.4# 
sh-4.4# 
sh-4.4# cd ..
sh-4.4# cd ..
sh-4.4# cd log
log/              logrotate-config/ 
sh-4.4# cd log/rbd-provisioner     
sh-4.4# ls
csi-attacher.log  csi-omap-generator.log  csi-provisioner.log  csi-snapshotter.log
sh-4.4# ls -lh
total 56K
-rw-r--r-- 1 root root 2.9K Jun 20 10:30 csi-attacher.log
-rw-r--r-- 1 root root 1.9K Jun 20 10:30 csi-omap-generator.log
-rw-r--r-- 1 root root 6.6K Jun 20 10:32 csi-provisioner.log
-rw-r--r-- 1 root root  37K Jun 20 10:33 csi-snapshotter.log
sh-4.4# truncate -s 2M csi-attacher.log 
sh-4.4# exit
exit
[rider@localhost examples]$ kubectl exec -it csi-rbdplugin-provisioner-5586ffc799-tg2pd -nrook-ceph -c log-collector sh
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
sh-4.4# cd /var/
adm/      crash/    empty/    games/    kerberos/ local/    log/      nis/      preserve/ spool/    tmp/      
cache/    db/       ftp/      gopher/   lib/      lock/     mail/     opt/      run/      target/   yp/       
sh-4.4# cd /var/lib/
NetworkManager/ cephadm/        dnf/            initramfs/      misc/           private/        rpm-state/      selinux/
alternatives/   cephcsi/        games/          kdump/          nfs/            rpcbind/        rsyslog/        smartmontools/
ceph/           dbus/           gssproxy/       logrotate/      portables/      rpm/            samba/          systemd/
sh-4.4# cd /var/lib/cephcsi/log                
sh-4.4# cd rbd-provisioner/    
sh-4.4# ls
csi-attacher.log       csi-omap-generator.log	    csi-provisioner.log       csi-snapshotter.log
csi-attacher.log.1.gz  csi-omap-generator.log.1.gz  csi-provisioner.log.1.gz  csi-snapshotter.log.1.gz

```
parth-gr: @BlaineEXE managed to fix all the CI issues, ready for final review and merge.
parth-gr: > Do we need changes for CephFS driver?
> 
> Do we need changes for NFS driver?

Open a separate PR, as separate testing is required

> Isn't this feature supposed to be enabled only when the CephCluster's log collection is enabled? The log flags on the templates are always-enabled from what I can tell.

Adding the `CSILogRotation` param to it
Madhu-1: it looks like as per the KEP https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/2845-deprecate-klog-specific-flags-in-k8s-components#writing-logs-to-files the klog is getting deprecated and these flags will be removed and an alternative is suggested for it. what is our take on this one? should we invest in this implementation or can we go for alternative suggestions as mentioned in the KEP.

Current discussion [here](https://github.com/kubernetes-csi/external-resizer/issues/411)

@travisn @BlaineEXE @parth-gr 
Madhu-1: @parth-gr we got a response from Kubernetes-CSI that we should not depend on these flags to write log to a file because those will go away soon and try to come up with something else.

BlaineEXE: > it looks like as per the KEP https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/2845-deprecate-klog-specific-flags-in-k8s-components#writing-logs-to-files the klog is getting deprecated and these flags will be removed and an alternative is suggested for it. what is our take on this one? should we invest in this implementation or can we go for alternative suggestions as mentioned in the KEP.

Thanks for bringing that up @Madhu-1 !

For me, as a maintainer, one of the goals I am always trying to balance is attempting to reduce the ways in which workload might be forced upon the team in the future. In this case, if we move forward with using klog flags for these sidecars, it sounds very likely that the team will have to deal with that in the future. We don't know or control when, and we also don't know how much time will be available for transitioning from klog flags, or even if go-runner will be made available in the containers CSI builds. That is an amount of risk and amount of likely future pain.

Comparing that risk/pain with the benefits, I personally think it would be best to not do log file collection and rotation for the sidecars at this time. I suggest that we do log collection/rotation for the applications that log the most (and most helpful) information for debugging Rook/CSI:

- Ceph-CSI provisioners
- Ceph-CSI plugins/drivers
- CSI Addons (Parth says this also logs a lot)

@Madhu-1 do you think this sounds like the right approach and best set of applications to focus on?
parth-gr: > @parth-gr we got a response from Kubernetes-CSI that we should not depend on these flags to write log to a file because those will go away soon and try to come up with something else.

Thats correct @Madhu-1 we had a discussion in the huddle today, Blaine said he will summarise his views,

But moreover in conclusion we had is, we can still use the same mechanism for the `ceph csi` container and the `csi-addons` sidecar and can remove the flag based file logging from other upstream sidecar containers,
And we believe the other side car containers doens't log that much, so we don;t need persist their log file and add log rotate,

Does that sounds fine to you?
Madhu-1: > > it looks like as per the KEP https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/2845-deprecate-klog-specific-flags-in-k8s-components#writing-logs-to-files the klog is getting deprecated and these flags will be removed and an alternative is suggested for it. what is our take on this one? should we invest in this implementation or can we go for alternative suggestions as mentioned in the KEP.
> 
> Thanks for bringing that up @Madhu-1 !
> 
> For me, as a maintainer, one of the goals I am always trying to balance is attempting to reduce the ways in which workload might be forced upon the team in the future. In this case, if we move forward with using klog flags for these sidecars, it sounds very likely that the team will have to deal with that in the future. We don't know or control when, and we also don't know how much time will be available for transitioning from klog flags, or even if go-runner will be made available in the containers CSI builds. That is an amount of risk and amount of likely future pain.
> 
> Comparing that risk/pain with the benefits, I personally think it would be best to not do log file collection and rotation for the sidecars at this time. I suggest that we do log collection/rotation for the applications that log the most (and most helpful) information for debugging Rook/CSI:
> 
> * Ceph-CSI provisioners
> * Ceph-CSI plugins/drivers
> * CSI Addons (Parth says this also logs a lot)
> 
> @Madhu-1 do you think this sounds like the right approach and best set of applications to focus on?

@BlaineEXE @parth-gr Thanks for the summary, Yes It sounds like a plan for the first phase, we can revisit other sidecar containers later on. 

Patch:
diff --git a/Documentation/CRDs/Cluster/external-cluster/external-cluster.md b/Documentation/CRDs/Cluster/external-cluster/external-cluster.md
index f57099175..378047371 100644
--- a/Documentation/CRDs/Cluster/external-cluster/external-cluster.md
+++ b/Documentation/CRDs/Cluster/external-cluster/external-cluster.md
@@ -63,32 +63,6 @@ python3 create-external-cluster-resources.py --rbd-data-pool-name <pool_name> --
 * `--topology-pools`: (optional) Comma-separated list of topology-constrained rbd pools
 * `--topology-failure-domain-label`: (optional) K8s cluster failure domain label (example: zone, rack, or host) for the topology-pools that match the ceph domain
 * `--topology-failure-domain-values`: (optional) Comma-separated list of the k8s cluster failure domain values corresponding to each of the pools in the `topology-pools` list
-* `--config-file`: Path to the configuration file, Priority: command-line-args > config.ini values > default values
-
-### Config-file
-
-Use the config file to set the user configuration file, add the flag `--config-file` to set the file path.
-
-Example:
-
-`/config.ini`
-
-```console
-[Configurations]
-format = bash
-cephfs-filesystem-name = <filesystem-name>
-rbd-data-pool-name = <pool_name>
-...
-```
-
-```console
-python3 create-external-cluster-resources.py --config-file /config.ini
-```
-
-!!! note
-    You can use both config file and other arguments at the same time
-    Priority: command-line-args > config.ini file values > default values
-
 
 ### Multi-tenancy
 
diff --git a/Documentation/Storage-Configuration/Shared-Filesystem-CephFS/filesystem-storage.md b/Documentation/Storage-Configuration/Shared-Filesystem-CephFS/filesystem-storage.md
index 930e1fcd8..ba6104671 100644
--- a/Documentation/Storage-Configuration/Shared-Filesystem-CephFS/filesystem-storage.md
+++ b/Documentation/Storage-Configuration/Shared-Filesystem-CephFS/filesystem-storage.md
@@ -391,7 +391,7 @@ kubectl delete -f kube-registry.yaml
 To delete the filesystem components and backing data, delete the Filesystem CRD.
 
 !!! warning
-    Data will be deleted if **preserveFilesystemOnDelete=false**.
+    Data will be deleted if preserveFilesystemOnDelete=false**.
 
 ```console
 kubectl -n rook-ceph delete cephfilesystem myfs
diff --git a/deploy/charts/rook-ceph/templates/resources.yaml b/deploy/charts/rook-ceph/templates/resources.yaml
index b0b909bd7..6727b5ba7 100644
--- a/deploy/charts/rook-ceph/templates/resources.yaml
+++ b/deploy/charts/rook-ceph/templates/resources.yaml
@@ -3556,6 +3556,7 @@ spec:
                             type: boolean
                           name:
                             description: Name is a unique identifier for the set
+                            maxLength: 50
                             type: string
                           placement:
                             nullable: true
diff --git a/deploy/examples/crds.yaml b/deploy/examples/crds.yaml
index bde3e25ef..70dfe726e 100644
--- a/deploy/examples/crds.yaml
+++ b/deploy/examples/crds.yaml
@@ -3554,6 +3554,7 @@ spec:
                             type: boolean
                           name:
                             description: Name is a unique identifier for the set
+                            maxLength: 50
                             type: string
                           placement:
                             nullable: true
diff --git a/deploy/examples/create-external-cluster-resources.py b/deploy/examples/create-external-cluster-resources.py
index cf2835c90..e0ccbcaf9 100644
--- a/deploy/examples/create-external-cluster-resources.py
+++ b/deploy/examples/create-external-cluster-resources.py
@@ -21,7 +21,6 @@ import argparse
 import re
 import subprocess
 import hmac
-import configparser
 from hashlib import sha1 as sha
 from os import linesep as LINESEP
 from os import path
@@ -479,15 +478,6 @@ class RadosJSON:
             + "Upgrade flag should only be used to append new permissions to users, it shouldn't be used for changing user already applied permission, for example you shouldn't change in which pool user has access",
         )
 
-        # Add command-line arguments
-        config_group = argP.add_argument_group("config")
-        config_group.add_argument(
-            "--config-file",
-            type=str,
-            help="Path to the configuration file, Priority: command-line-args > config.ini file values > default values",
-            required=False,
-        )
-
         if args_to_parse:
             assert (
                 type(args_to_parse) == list
@@ -627,31 +617,10 @@ class RadosJSON:
             ("-1"),
         )
 
-    def parse_config_file(self, config_file):
-        config = configparser.ConfigParser()
-        config.read(config_file)
-        for arg in list(vars(self._arg_parser)):
-            # python treats flag-name as flag_name internally, so converting back to flag-name,
-            # so we can get those values from config file
-            argument = arg.replace("_", "-")
-            argumentValue = str(getattr(self._arg_parser, arg))
-            config_value = config.get("Configurations", argument, fallback=None)
-            # give priority to command line argument, if command line argument is not present use config.ini value,
-            if (
-                (str(sys.argv).find(argument) == -1)
-                and ((argumentValue is None) or (argumentValue is ""))
-                and ((config_value is not None) and (config_value is not ""))
-            ):
-                self._arg_parser.__setattr__(arg, config_value)
-
-        return config
-
     def __init__(self, arg_list=None):
         self.out_map = {}
         self._excluded_keys = set()
         self._arg_parser = self.gen_arg_parser(args_to_parse=arg_list)
-        if self._arg_parser.config_file:
-            self.config = self.parse_config_file(self._arg_parser.config_file)
         self._check_conflicting_options()
         self.run_as_user = self._arg_parser.run_as_user
         self.output_file = self._arg_parser.output
@@ -1559,9 +1528,6 @@ class RadosJSON:
         for pool in topology_rbd_pools:
             self.init_rbd_pool(pool)
 
-    def getScriptCliFlags(self):
-        return " ".join(sys.argv[1:])
-
     def _gen_output_map(self):
         if self.out_map:
             return
@@ -1576,7 +1542,6 @@ class RadosJSON:
         self.validate_rados_namespace()
         self._excluded_keys.add("K8S_CLUSTER_NAME")
         self.get_cephfs_data_pool_details()
-        self.out_map["EXTERNAL_CLUSTER_USER_COMMAND"] = self.getScriptCliFlags()
         self.out_map["NAMESPACE"] = self._arg_parser.namespace
         self.out_map["K8S_CLUSTER_NAME"] = self._arg_parser.k8s_cluster_name
         self.out_map["ROOK_EXTERNAL_FSID"] = self.get_fsid()
@@ -1737,13 +1702,6 @@ class RadosJSON:
         if self._arg_parser.dry_run:
             return ""
         json_out = [
-            {
-                "name": "external-cluster-user-command",
-                "kind": "ConfigMap",
-                "data": {
-                    "command": self.out_map["EXTERNAL_CLUSTER_USER_COMMAND"],
-                },
-            },
             {
                 "name": "rook-ceph-mon-endpoints",
                 "kind": "ConfigMap",
diff --git a/deploy/examples/import-external-cluster.sh b/deploy/examples/import-external-cluster.sh
index d378229f0..c9afb003a 100644
--- a/deploy/examples/import-external-cluster.sh
+++ b/deploy/examples/import-external-cluster.sh
@@ -14,7 +14,6 @@ MON_SECRET_MON_KEYRING_KEYNAME=mon-secret
 MON_SECRET_CEPH_USERNAME_KEYNAME=ceph-username
 MON_SECRET_CEPH_SECRET_KEYNAME=ceph-secret
 MON_ENDPOINT_CONFIGMAP_NAME=rook-ceph-mon-endpoints
-EXTERNAL_COMMAND_CONFIGMAP_NAME=external-cluster-user-command
 ROOK_EXTERNAL_CLUSTER_NAME=$NAMESPACE
 ROOK_RBD_FEATURES=${ROOK_RBD_FEATURES:-"layering"}
 ROOK_EXTERNAL_MAX_MON_ID=2
@@ -168,23 +167,6 @@ function importConfigMap() {
   fi
 }
 
-function createInputCommadConfigMap() {
-  if ! $KUBECTL -n "$NAMESPACE" get configmap "$EXTERNAL_COMMAND_CONFIGMAP_NAME" &>/dev/null; then
-    $KUBECTL -n "$NAMESPACE" \
-      create \
-      configmap \
-      "$EXTERNAL_COMMAND_CONFIGMAP_NAME" \
-      --from-literal=command="$EXTERNAL_CLUSTER_USER_COMMAND"
-  else
-    echo "configmap $EXTERNAL_COMMAND_CONFIGMAP_NAME already exists, updating it"
-    $KUBECTL -n "$NAMESPACE" \
-      patch \
-      configmap \
-      "$EXTERNAL_COMMAND_CONFIGMAP_NAME" \
-      -p "{'data':{'command':$EXTERNAL_COMMAND_CONFIGMAP_NAME}}"
-  fi
-}
-
 function importCsiRBDNodeSecret() {
   if ! $KUBECTL -n "$NAMESPACE" get secret "rook-$CSI_RBD_NODE_SECRET_NAME" &>/dev/null; then
     $KUBECTL -n "$NAMESPACE" \
@@ -407,7 +389,6 @@ createClusterNamespace
 importClusterID
 importSecret
 importConfigMap
-createInputCommadConfigMap
 if [ -n "$CSI_RBD_NODE_SECRET_NAME" ] && [ -n "$CSI_RBD_NODE_SECRET" ]; then
   importCsiRBDNodeSecret
 fi
diff --git a/go.mod b/go.mod
index d5b8a98da..0198f510c 100644
--- a/go.mod
+++ b/go.mod
@@ -14,7 +14,7 @@ replace (
 
 require (
 	github.com/IBM/keyprotect-go-client v0.14.1
-	github.com/aws/aws-sdk-go v1.54.6
+	github.com/aws/aws-sdk-go v1.54.2
 	github.com/banzaicloud/k8s-objectmatcher v1.8.0
 	github.com/ceph/go-ceph v0.28.0
 	github.com/coreos/pkg v0.0.0-20230601102743-20bbbf26f4d8
diff --git a/go.sum b/go.sum
index fe9effcd9..2cbb0e3a2 100644
--- a/go.sum
+++ b/go.sum
@@ -144,8 +144,8 @@ github.com/armon/go-socks5 v0.0.0-20160902184237-e75332964ef5/go.mod h1:wHh0iHkY
 github.com/asaskevich/govalidator v0.0.0-20180720115003-f9ffefc3facf/go.mod h1:lB+ZfQJz7igIIfQNfa7Ml4HSf2uFQQRzpGGRXenZAgY=
 github.com/asaskevich/govalidator v0.0.0-20190424111038-f61b66f89f4a/go.mod h1:lB+ZfQJz7igIIfQNfa7Ml4HSf2uFQQRzpGGRXenZAgY=
 github.com/aws/aws-sdk-go v1.44.164/go.mod h1:aVsgQcEevwlmQ7qHE9I3h+dtQgpqhFB+i8Phjh7fkwI=
-github.com/aws/aws-sdk-go v1.54.6 h1:HEYUib3yTt8E6vxjMWM3yAq5b+qjj/6aKA62mkgux9g=
-github.com/aws/aws-sdk-go v1.54.6/go.mod h1:eRwEWoyTWFMVYVQzKMNHWP5/RV4xIUGMQfXQHfHkpNU=
+github.com/aws/aws-sdk-go v1.54.2 h1:Wo6AVWcleNHrYa48YzfYz60hzxGRqsJrK5s/qePe+3I=
+github.com/aws/aws-sdk-go v1.54.2/go.mod h1:eRwEWoyTWFMVYVQzKMNHWP5/RV4xIUGMQfXQHfHkpNU=
 github.com/banzaicloud/k8s-objectmatcher v1.8.0 h1:Nugn25elKtPMTA2br+JgHNeSQ04sc05MDPmpJnd1N2A=
 github.com/banzaicloud/k8s-objectmatcher v1.8.0/go.mod h1:p2LSNAjlECf07fbhDyebTkPUIYnU05G+WfGgkTmgeMg=
 github.com/benbjohnson/clock v1.1.0/go.mod h1:J11/hYXuz8f4ySSvYwY0FKfm+ezbsZBKZxNJlLklBHA=
diff --git a/images/ceph/Makefile b/images/ceph/Makefile
index 10f9e9091..77570c1f6 100755
--- a/images/ceph/Makefile
+++ b/images/ceph/Makefile
@@ -60,6 +60,7 @@ do.build:
 	@cp -r $(MANIFESTS_DIR)/monitoring $(BUILD_CONTEXT_DIR)/ceph-monitoring
 	@mkdir -p $(BUILD_CONTEXT_DIR)/rook-external/test-data
 	@cp $(MANIFESTS_DIR)/create-external-cluster-resources.* $(BUILD_CONTEXT_DIR)/rook-external/
+	@cp ../../tests/ceph-status-out $(BUILD_CONTEXT_DIR)/rook-external/test-data/
 	@cd $(BUILD_CONTEXT_DIR) && $(SED_IN_PLACE) 's|BASEIMAGE|$(BASEIMAGE)|g' Dockerfile
 	@if [ -z "$(BUILD_CONTAINER_IMAGE)" ]; then\
 		$(DOCKERCMD) build $(BUILD_ARGS) \
diff --git a/pkg/apis/ceph.rook.io/v1/types.go b/pkg/apis/ceph.rook.io/v1/types.go
index 8a3762573..abd64d7c4 100755
--- a/pkg/apis/ceph.rook.io/v1/types.go
+++ b/pkg/apis/ceph.rook.io/v1/types.go
@@ -2968,6 +2968,7 @@ type PriorityClassNamesSpec map[KeyType]string
 // +nullable
 type StorageClassDeviceSet struct {
 	// Name is a unique identifier for the set
+	// +kubebuilder:validation:MaxLength=50
 	Name string `json:"name"`
 	// Count is the number of devices in this set
 	// +kubebuilder:validation:Minimum=1
diff --git a/pkg/daemon/ceph/osd/encryption.go b/pkg/daemon/ceph/osd/encryption.go
index 2c83c64dd..785d0a363 100644
--- a/pkg/daemon/ceph/osd/encryption.go
+++ b/pkg/daemon/ceph/osd/encryption.go
@@ -272,10 +272,10 @@ func addEncryptionKey(context *clusterd.Context, disk, passphrase, newPassphrase
 		if err != nil {
 			return errors.Wrapf(err, "failed to ensure passphrase in slot %q of encrypted device %q", slot, disk)
 		}
-		// if newPassphrase is not one in the slot, then remove the key slot using current passphrase and then
-		// add the newPassphrase to it.
+		// if newPassphrase is not one in the slot, then remove the key slot and
+		// add add the newPassphrase to it.
 		if !matched {
-			err = removeEncryptionKeySlot(context, disk, passphrase, slot)
+			err = removeEncryptionKeySlot(context, disk, newPassphrase, slot)
 			if err != nil {
 				return errors.Wrapf(err, "failed to remove key slot %q of encrypted device %q", slot, disk)
 			}
diff --git a/pkg/operator/ceph/cluster/osd/deviceSet.go b/pkg/operator/ceph/cluster/osd/deviceSet.go
index 0b987212b..fbbfa9080 100644
--- a/pkg/operator/ceph/cluster/osd/deviceSet.go
+++ b/pkg/operator/ceph/cluster/osd/deviceSet.go
@@ -205,16 +205,20 @@ func (c *Cluster) createDeviceSetPVC(existingPVCs map[string]*v1.PersistentVolum
 	// old labels and PVC ID for backward compatibility
 	pvcID := legacyDeviceSetPVCID(deviceSetName, setIndex)
 
+	var err error
 	// check for the existence of the pvc
 	existingPVC, ok := existingPVCs[pvcID]
 	if !ok {
 		// The old name of the PVC didn't exist, now try the new PVC name and label
-		pvcID = deviceSetPVCID(deviceSetName, pvcTemplate.GetName(), setIndex)
+		pvcID, err = deviceSetPVCID(deviceSetName, pvcTemplate.GetName(), setIndex)
+		if err != nil {
+			return nil, err
+		}
 		existingPVC = existingPVCs[pvcID]
 	}
 
 	pvc := makeDeviceSetPVC(deviceSetName, pvcID, setIndex, pvcTemplate, c.clusterInfo.Namespace, createValidImageVersionLabel(c.spec.CephVersion.Image), createValidImageVersionLabel(c.rookVersion))
-	err := c.clusterInfo.OwnerInfo.SetControllerReference(pvc)
+	err = c.clusterInfo.OwnerInfo.SetControllerReference(pvc)
 	if err != nil {
 		return nil, errors.Wrapf(err, "failed to set owner reference to osd pvc %q", pvc.Name)
 	}
@@ -291,10 +295,14 @@ func legacyDeviceSetPVCID(deviceSetName string, setIndex int) string {
 
 // This is the new function that generates the labels
 // It includes the pvcTemplateName in it
-func deviceSetPVCID(deviceSetName, pvcTemplateName string, setIndex int) string {
+func deviceSetPVCID(deviceSetName, pvcTemplateName string, setIndex int) (string, error) {
 	cleanName := strings.Replace(pvcTemplateName, " ", "-", -1)
 	deviceSetName = strings.Replace(deviceSetName, ".", "-", -1)
-	return fmt.Sprintf("%s-%s-%d", deviceSetName, cleanName, setIndex)
+	pvcID := fmt.Sprintf("%s-%s-%d", deviceSetName, cleanName, setIndex)
+	if len(pvcID) > 62 {
+		return "", fmt.Errorf("the OSD PVC name requested is %q (length %d) is too long and must be less than 63 characters, shorten either the storageClassDeviceSet name or the storage class name", pvcID, len(pvcID))
+	}
+	return pvcID, nil
 }
 
 func createValidImageVersionLabel(image string) string {
diff --git a/pkg/operator/ceph/cluster/osd/spec.go b/pkg/operator/ceph/cluster/osd/spec.go
index bf8066e04..a2285f062 100644
--- a/pkg/operator/ceph/cluster/osd/spec.go
+++ b/pkg/operator/ceph/cluster/osd/spec.go
@@ -183,10 +183,7 @@ sys.exit('no disk found with OSD ID $OSD_ID')
 "
 	}
 
-	if ! ceph-volume raw list "$DEVICE" > "$OSD_LIST"; then
-		# if the command fails, the disk may be renamed
-		echo '' > "$OSD_LIST"
-	fi
+	ceph-volume raw list "$DEVICE" > "$OSD_LIST"
 	cat "$OSD_LIST"
 
 	if ! find_device < "$OSD_LIST"; then
diff --git a/pkg/operator/ceph/cluster/watcher.go b/pkg/operator/ceph/cluster/watcher.go
index a79c71221..f16b0458e 100644
--- a/pkg/operator/ceph/cluster/watcher.go
+++ b/pkg/operator/ceph/cluster/watcher.go
@@ -168,16 +168,6 @@ func (c *clientCluster) handleNodeFailure(ctx context.Context, cluster *cephv1.C
 		return nil
 	}
 
-	disabledCSI, err := k8sutil.GetOperatorSetting(ctx, c.context.Clientset, opcontroller.OperatorSettingConfigMapName, "ROOK_CSI_DISABLE_DRIVER", "false")
-	if err != nil {
-		return pkgerror.Wrapf(err, "failed to get configmap value `ROOK_CSI_DISABLE_DRIVER`.")
-	}
-
-	if strings.ToLower(disabledCSI) != "false" {
-		logger.Debugf("not watching for node failures since `ROOK_CSI_DISABLE_DRIVER` is set to %q, skip creating networkFence", disabledCSI)
-		return nil
-	}
-
 	_, err = c.context.ApiExtensionsClient.ApiextensionsV1().CustomResourceDefinitions().Get(ctx, "networkfences.csiaddons.openshift.io", metav1.GetOptions{})
 	if err != nil {
 		if errors.IsNotFound(err) {
diff --git a/pkg/operator/ceph/controller/spec.go b/pkg/operator/ceph/controller/spec.go
index 9a2afa27b..73c5a424e 100644
--- a/pkg/operator/ceph/controller/spec.go
+++ b/pkg/operator/ceph/controller/spec.go
@@ -757,9 +757,7 @@ func PrivilegedContext(runAsRoot bool) *v1.SecurityContext {
 	return sec
 }
 
-// LogCollectorContainer rotate logs
-func LogCollectorContainer(daemonID, ns string, c cephv1.ClusterSpec) *v1.Container {
-
+func GetLogRotateConfig(c cephv1.ClusterSpec) (resource.Quantity, string) {
 	var maxLogSize resource.Quantity
 	if c.LogCollector.MaxLogSize != nil {
 		size := c.LogCollector.MaxLogSize.Value() / 1000 / 1000
@@ -771,20 +769,28 @@ func LogCollectorContainer(daemonID, ns string, c cephv1.ClusterSpec) *v1.Contai
 		maxLogSize = resource.MustParse(fmt.Sprintf("%dM", size))
 	}
 
-	rotation := "7"
-	if strings.Contains(daemonID, "-client.rbd-mirror") {
-		rotation = "28"
-	}
-
 	var periodicity string
-	if c.LogCollector.Periodicity == "1h" || c.LogCollector.Periodicity == "hourly" {
+	switch c.LogCollector.Periodicity {
+	case "1h", "hourly":
 		periodicity = "hourly"
-	} else if c.LogCollector.Periodicity == "weekly" || c.LogCollector.Periodicity == "monthly" {
+	case "weekly", "monthly":
 		periodicity = c.LogCollector.Periodicity
-	} else {
+	default:
 		periodicity = "daily"
 	}
 
+	return maxLogSize, periodicity
+}
+
+// LogCollectorContainer rotate logs
+func LogCollectorContainer(daemonID, ns string, c cephv1.ClusterSpec) *v1.Container {
+	maxLogSize, periodicity := GetLogRotateConfig(c)
+
+	rotation := "7"
+	if strings.Contains(daemonID, "-client.rbd-mirror") {
+		rotation = "28"
+	}
+
 	logger.Debugf("setting periodicity to %q. Supported periodicity are hourly, daily, weekly and monthly", periodicity)
 
 	return &v1.Container{
diff --git a/pkg/operator/ceph/csi/controller.go b/pkg/operator/ceph/csi/controller.go
index 87070a2a4..467791a11 100644
--- a/pkg/operator/ceph/csi/controller.go
+++ b/pkg/operator/ceph/csi/controller.go
@@ -205,7 +205,7 @@ func (r *ReconcileCSI) reconcile(request reconcile.Request) (reconcile.Result, e
 		return opcontroller.ImmediateRetryResult, errors.Wrap(err, "failed to list ceph clusters")
 	}
 
-	// // Do not nothing if no ceph cluster is present
+	// Do nothing if no ceph cluster is present
 	if len(cephClusters.Items) == 0 {
 		logger.Debug("no ceph cluster found not deploying ceph csi driver")
 		EnableRBD, EnableCephFS, EnableNFS = false, false, false
@@ -217,6 +217,10 @@ func (r *ReconcileCSI) reconcile(request reconcile.Request) (reconcile.Result, e
 		return reconcile.Result{}, nil
 	}
 
+	// if at least one cephcluster is present update the csi lograte sidecar
+	// with the first listed ceph cluster specs with logrotate enabled
+	setCSILogrotateParams(cephClusters.Items)
+
 	err = peermap.CreateOrUpdateConfig(r.opManagerContext, r.context, &peermap.PeerIDMappings{})
 	if err != nil {
 		return opcontroller.ImmediateRetryResult, errors.Wrap(err, "failed to create pool ID mapping config map")
@@ -299,3 +303,24 @@ func (r *ReconcileCSI) reconcile(request reconcile.Request) (reconcile.Result, e
 
 	return reconcileResult, nil
 }
+
+func setCSILogrotateParams(cephClustersItems []cephv1.CephCluster) {
+	logger.Debug("set logrotate values in csi param")
+	spec := cephClustersItems[0].Spec
+	for _, cluster := range cephClustersItems {
+		if cluster.Spec.LogCollector.Enabled {
+			spec = cluster.Spec
+			break
+		}
+	}
+	CSIParam.CsiLogDirPath = spec.DataDirHostPath
+	if spec.DataDirHostPath == "" {
+		CSIParam.CsiLogDirPath = k8sutil.DataDir
+	}
+	CSIParam.CSILogRotation = spec.LogCollector.Enabled
+	if spec.LogCollector.Enabled {
+		maxSize, period := opcontroller.GetLogRotateConfig(spec)
+		CSIParam.CSILogRotationMaxSize = maxSize.String()
+		CSIParam.CSILogRotationPeriod = period
+	}
+}
diff --git a/pkg/operator/ceph/csi/spec.go b/pkg/operator/ceph/csi/spec.go
index 70bf0c6f4..a02a98338 100644
--- a/pkg/operator/ceph/csi/spec.go
+++ b/pkg/operator/ceph/csi/spec.go
@@ -52,6 +52,7 @@ type Param struct {
 	ResizerImage                             string
 	DriverNamePrefix                         string
 	KubeletDirPath                           string
+	CsiLogDirPath                            string
 	ForceCephFSKernelClient                  string
 	CephFSKernelMountOptions                 string
 	CephFSPluginUpdateStrategy               string
@@ -97,6 +98,10 @@ type Param struct {
 	CSICephFSPodLabels                       map[string]string
 	CSINFSPodLabels                          map[string]string
 	CSIRBDPodLabels                          map[string]string
+	CSILogRotation                           bool
+	CSILogRotationMaxSize                    string
+	CSILogRotationPeriod                     string
+	CSILogFolder                             string
 }
 
 type templateParam struct {
@@ -178,6 +183,9 @@ var (
 	//go:embed template/nfs/csi-nfsplugin-holder.yaml
 	NFSPluginHolderTemplatePath string
 
+	//go:embed template/csi-logrotate-sidecar.yaml
+	LogrotateTemplatePath string
+
 	holderEnabled bool
 )
 
@@ -349,11 +357,19 @@ func (r *ReconcileCSI) startDrivers(ver *version.Info, ownerInfo *k8sutil.OwnerI
 		if err != nil {
 			return errors.Wrap(err, "failed to load rbdplugin template")
 		}
+		if tp.CSILogRotation {
+			tp.CSILogFolder = "rbd-plugin"
+			applyLogrotateSidecar(&rbdPlugin.Spec.Template, "csi-rbd-daemonset-log-collector", LogrotateTemplatePath, tp)
+		}
 
 		rbdProvisionerDeployment, err = templateToDeployment("rbd-provisioner", RBDProvisionerDepTemplatePath, tp)
 		if err != nil {
 			return errors.Wrap(err, "failed to load rbd provisioner deployment template")
 		}
+		if tp.CSILogRotation {
+			tp.CSILogFolder = "rbd-provisioner"
+			applyLogrotateSidecar(&rbdProvisionerDeployment.Spec.Template, "csi-rbd-deployment-log-collector", LogrotateTemplatePath, tp)
+		}
 
 		// Create service if either liveness or GRPC metrics are enabled.
 		if CSIParam.EnableLiveness {
diff --git a/pkg/operator/ceph/csi/template/csi-logrotate-sidecar.yaml b/pkg/operator/ceph/csi/template/csi-logrotate-sidecar.yaml
new file mode 100644
index 000000000..63a47286c
--- /dev/null
+++ b/pkg/operator/ceph/csi/template/csi-logrotate-sidecar.yaml
@@ -0,0 +1,35 @@
+args:
+  - |
+    echo "Starting the csi-logrotate-sidecar"
+    mkdir -p {{ .CsiLogDirPath }}/cephcsi/logrotate-config/{{ .CSILogFolder }};
+    echo '{{ .CsiLogDirPath }}cephcsi/log/{{ .CSILogFolder }}/*.log {
+        {{ .CSILogRotationPeriod }}
+        missingok
+        rotate 7
+        compress
+        copytruncate
+        notifempty
+    }' >  {{ .CsiLogDirPath }}/cephcsi/logrotate-config/{{ .CSILogFolder }}/csi;
+    echo "File creation container completed";
+
+    LOG_ROTATE_CEPH_CSI_FILE={{ .CsiLogDirPath }}/cephcsi/logrotate-config/{{ .CSILogFolder }}/csi
+    LOG_MAX_SIZE={{ .CSILogRotationMaxSize }}
+    if [ "$LOG_MAX_SIZE" != "0" ]; then
+      sed --in-place "4i \ \ \ \ maxsize $LOG_MAX_SIZE" "$LOG_ROTATE_CEPH_CSI_FILE"
+    fi
+
+    while true; do
+      logrotate --verbose "$LOG_ROTATE_CEPH_CSI_FILE"
+      sleep 15m
+    done
+command:
+  - /bin/sh
+  - -c
+image: {{ .CSIPluginImage }}
+imagePullPolicy: IfNotPresent
+name: log-collector
+volumeMounts:
+  - mountPath: {{ .CsiLogDirPath }}/cephcsi/logrotate-config/{{ .CSILogFolder }}
+    name: csi-logs-logrotate
+  - mountPath: {{ .CsiLogDirPath }}/cephcsi/log/{{ .CSILogFolder }}
+    name: csi-log
diff --git a/pkg/operator/ceph/csi/template/rbd/csi-rbdplugin-provisioner-dep.yaml b/pkg/operator/ceph/csi/template/rbd/csi-rbdplugin-provisioner-dep.yaml
index 05dc8bfc4..2265b0182 100644
--- a/pkg/operator/ceph/csi/template/rbd/csi-rbdplugin-provisioner-dep.yaml
+++ b/pkg/operator/ceph/csi/template/rbd/csi-rbdplugin-provisioner-dep.yaml
@@ -38,9 +38,9 @@ spec:
             - "--extra-create-metadata=true"
             - "--prevent-volume-mode-conversion=true"
             - "--feature-gates=HonorPVReclaimPolicy=true"
-            {{- if .EnableCSITopology }}
+            {{ if .EnableCSITopology }}
             - "--feature-gates=Topology=true"
-            {{- end }}
+            {{ end }}
           env:
             - name: ADDRESS
               value: unix:///csi/csi-provisioner.sock
@@ -116,7 +116,7 @@ spec:
         {{ if .EnableOMAPGenerator }}
         - name: csi-omap-generator
           image: {{ .CSIPluginImage }}
-          args :
+          args:
             - "--type=controller"
             - "--drivernamespace=$(DRIVER_NAMESPACE)"
             - "--v={{ .LogLevel }}"
@@ -142,7 +142,7 @@ spec:
         {{ if .EnableCSIAddonsSideCar }}
         - name: csi-addons
           image: {{ .CSIAddonsImage }}
-          args :
+          args:
             - "--node-id=$(NODE_ID)"
             - "--v={{ .LogLevel }}"
             - "--csi-addons-address=$(CSIADDONS_ENDPOINT)"
@@ -155,6 +155,11 @@ spec:
             - "--leader-election-lease-duration={{ .LeaderElectionLeaseDuration }}"
             - "--leader-election-renew-deadline={{ .LeaderElectionRenewDeadline }}"
             - "--leader-election-retry-period={{ .LeaderElectionRetryPeriod }}"
+            {{ if .CSILogRotation }}
+            - "--logtostderr=false"
+            - "--alsologtostderr=true"
+            - "--log_file={{ .CsiLogDirPath }}/cephcsi/log/rbd-provisioner/csi-addons.log"
+            {{ end }}
           ports:
             - containerPort: {{ .CSIAddonsPort }}
           env:
@@ -180,10 +185,14 @@ spec:
           volumeMounts:
             - name: socket-dir
               mountPath: /csi
+            {{ if .CSILogRotation }}
+            - mountPath: {{ .CsiLogDirPath }}/cephcsi/log/rbd-provisioner
+              name: csi-log
+            {{ end }}
         {{ end }}
         - name: csi-rbdplugin
           image: {{ .CSIPluginImage }}
-          args :
+          args:
             - "--nodeid=$(NODE_ID)"
             - "--endpoint=$(CSI_ENDPOINT)"
             - "--v={{ .LogLevel }}"
@@ -191,6 +200,11 @@ spec:
             - "--controllerserver=true"
             - "--drivername={{ .DriverNamePrefix }}rbd.csi.ceph.com"
             - "--pidlimit=-1"
+            {{ if .CSILogRotation }}
+            - "--logtostderr=false"
+            - "--alsologtostderr=true"
+            - "--log_file={{ .CsiLogDirPath }}/cephcsi/log/rbd-provisioner/csi-rbdplugin.log"
+            {{ end }}
             {{ if .EnableCSIAddonsSideCar }}
             - "--csi-addons-endpoint=$(CSIADDONS_ENDPOINT)"
             {{ end }}
@@ -225,6 +239,10 @@ spec:
               mountPath: /csi
             - mountPath: /dev
               name: host-dev
+            {{ if .CSILogRotation }}
+            - mountPath: {{ .CsiLogDirPath }}/cephcsi/log/rbd-provisioner
+              name: csi-log
+            {{ end }}
             - mountPath: /sys
               name: host-sys
             - mountPath: /lib/modules
@@ -272,6 +290,13 @@ spec:
         - name: host-dev
           hostPath:
             path: /dev
+        - name: csi-log
+          hostPath:
+            path: {{ .CsiLogDirPath }}/cephcsi/log/rbd-provisioner
+            type: DirectoryOrCreate
+        - name: csi-logs-logrotate
+          emptyDir:
+            type: DirectoryOrCreate
         - name: host-sys
           hostPath:
             path: /sys
diff --git a/pkg/operator/ceph/csi/template/rbd/csi-rbdplugin.yaml b/pkg/operator/ceph/csi/template/rbd/csi-rbdplugin.yaml
index 960598ace..733cfa2dd 100644
--- a/pkg/operator/ceph/csi/template/rbd/csi-rbdplugin.yaml
+++ b/pkg/operator/ceph/csi/template/rbd/csi-rbdplugin.yaml
@@ -73,6 +73,11 @@ spec:
             - "--drivername={{ .DriverNamePrefix }}rbd.csi.ceph.com"
             - "--pidlimit=-1"
             - "--stagingpath={{ .KubeletDirPath }}/plugins/kubernetes.io/csi/"
+            {{ if .CSILogRotation }}
+            - "--logtostderr=false"
+            - "--alsologtostderr=true"
+            - "--log_file={{ .CsiLogDirPath }}/cephcsi/log/rbd-plugin/csi-rbdplugin.log"
+            {{ end }}
             {{ if .EnableCSIAddonsSideCar }}
             - "--csi-addons-endpoint=$(CSIADDONS_ENDPOINT)"
             {{ end }}
@@ -121,6 +126,10 @@ spec:
               mountPath: /tmp/csi/keys
             - name: host-run-mount
               mountPath: /run/mount
+            {{ if .CSILogRotation }}
+            - mountPath: {{ .CsiLogDirPath }}/cephcsi/log/rbd-plugin
+              name: csi-log
+            {{ end }}
             {{ if .EnablePluginSelinuxHostMount }}
             - name: etc-selinux
               mountPath: /etc/selinux
@@ -154,6 +163,11 @@ spec:
             - "--namespace=$(POD_NAMESPACE)"
             - "--pod-uid=$(POD_UID)"
             - "--stagingpath={{ .KubeletDirPath }}/plugins/kubernetes.io/csi/"
+            {{ if .CSILogRotation }}
+            - "--logtostderr=false"
+            - "--alsologtostderr=true"
+            - "--log_file={{ .CsiLogDirPath }}/cephcsi/log/rbd-plugin/csi-addons.log"
+            {{ end }}
           ports:
             - containerPort: {{ .CSIAddonsPort }}
           env:
@@ -179,6 +193,10 @@ spec:
           volumeMounts:
             - name: plugin-dir
               mountPath: /csi
+            {{ if .CSILogRotation }}
+            - mountPath: {{ .CsiLogDirPath }}/cephcsi/log/rbd-plugin
+              name: csi-log
+            {{ end }}
         {{ end }}
         {{ if .EnableLiveness }}
         - name: liveness-prometheus
@@ -211,6 +229,13 @@ spec:
           hostPath:
             path: "{{ .KubeletDirPath }}/plugins/{{ .DriverNamePrefix }}rbd.csi.ceph.com"
             type: DirectoryOrCreate
+        - name: csi-log
+          hostPath:
+            path: {{ .CsiLogDirPath }}/cephcsi/log/rbd-plugin
+            type: DirectoryOrCreate
+        - name: csi-logs-logrotate
+          emptyDir:
+            type: DirectoryOrCreate
         - name: plugin-mount-dir
           hostPath:
             path: "{{ .KubeletDirPath }}/plugins"
diff --git a/pkg/operator/ceph/csi/util.go b/pkg/operator/ceph/csi/util.go
index 0eddca95a..f2b4c379c 100644
--- a/pkg/operator/ceph/csi/util.go
+++ b/pkg/operator/ceph/csi/util.go
@@ -84,6 +84,20 @@ func templateToDeployment(name, templateData string, p templateParam) (*apps.Dep
 	return &dep, nil
 }
 
+func applyLogrotateSidecar(specTemplate *corev1.PodTemplateSpec, name, templateData string, p templateParam) {
+	var logrotateSidecarContainer corev1.Container
+	t, err := loadTemplate(name, templateData, p)
+	if err != nil {
+		panic(errors.Wrap(err, "failed to load logrotate container template"))
+	}
+
+	err = yaml.Unmarshal(t, &logrotateSidecarContainer)
+	if err != nil {
+		panic(errors.Wrap(err, "failed to unmarshal logrotate container template"))
+	}
+	specTemplate.Spec.Containers = append(specTemplate.Spec.Containers, logrotateSidecarContainer)
+}
+
 func applyResourcesToContainers(opConfig map[string]string, key string, podspec *corev1.PodSpec) {
 	resource := getComputeResource(opConfig, key)
 


Test patch:
diff --git a/.github/workflows/canary-integration-test.yml b/.github/workflows/canary-integration-test.yml
index b68dcbb37..3da8fdbe2 100644
--- a/.github/workflows/canary-integration-test.yml
+++ b/.github/workflows/canary-integration-test.yml
@@ -68,8 +68,7 @@ jobs:
         run: |
           toolbox=$(kubectl get pod -l app=rook-ceph-tools -n rook-ceph -o jsonpath='{.items[*].metadata.name}')
           kubectl -n rook-ceph exec $toolbox -- mkdir -p /etc/ceph/test-data
-          kubectl -n rook-ceph cp tests/external-cluster/ceph-status-out $toolbox:/etc/ceph/test-data/
-          kubectl -n rook-ceph cp tests/external-cluster/external-config.ini $toolbox:/etc/ceph/
+          kubectl -n rook-ceph cp tests/ceph-status-out $toolbox:/etc/ceph/test-data/
           kubectl -n rook-ceph cp deploy/examples/create-external-cluster-resources.py $toolbox:/etc/ceph
           kubectl -n rook-ceph cp deploy/examples/create-external-cluster-resources-tests.py $toolbox:/etc/ceph
           timeout 10 sh -c "until kubectl -n rook-ceph exec $toolbox -- python3 /etc/ceph/create-external-cluster-resources.py --rbd-data-pool-name replicapool; do echo 'waiting for script to succeed' && sleep 1; done"
@@ -268,29 +267,6 @@ jobs:
           toolbox=$(kubectl get pod -l app=rook-ceph-tools -n rook-ceph -o jsonpath='{.items[*].metadata.name}')
           kubectl -n rook-ceph exec $toolbox -- python3 /etc/ceph/create-external-cluster-resources.py --rbd-data-pool-name replicapool --v2-port-enable
 
-      - name: test config file flag
-        run: |
-          toolbox=$(kubectl get pod -l app=rook-ceph-tools -n rook-ceph -o jsonpath='{.items[*].metadata.name}')
-          # create `radosNamespace2` rados-namespace for `replicapool` rbd data-pool
-          kubectl -n rook-ceph exec $toolbox -- rbd namespace create replicapool/radosnamespace2
-
-          # validate `--rados-namespace`, config file value is radosnamespace2, cli arg flag is set to radosnamespace1 and no default is set
-          # command line arg should be used
-          kubectl -n rook-ceph exec $toolbox -- python3 /etc/ceph/create-external-cluster-resources.py --rbd-data-pool-name replicapool --rados-namespace radosnamespace1 --config-file /etc/ceph/external-config.ini
-
-          # validate the `rbd-data-pool-name`,config value set to replicapoolconfig, command line arg is not set and no default
-          # config value should be used
-          sed -i 's/replicapool/replicapoolconfig/' deploy/examples/pool-test.yaml
-          kubectl create -f deploy/examples/pool-test.yaml
-          kubectl wait --for='jsonpath={.status.phase}=Ready' Cephblockpool/replicapoolconfig -nrook-ceph
-          # create `radosNamespace2` rados-namespace for `replicapoolconfig` rbd data-pool
-          kubectl -n rook-ceph exec $toolbox -- rbd namespace create replicapoolconfig/radosnamespace2
-          kubectl -n rook-ceph exec $toolbox -- python3 /etc/ceph/create-external-cluster-resources.py --config-file /etc/ceph/external-config.ini
-
-          # validate the default `format`` flag, config file value is not set and no cmd line argument is set and default value is `Json``,
-          # default value should be used
-          kubectl -n rook-ceph exec $toolbox -- python3 /etc/ceph/create-external-cluster-resources.py --rbd-data-pool-name replicapool --config-file /etc/ceph/external-config.ini
-
       - name: check-ownerreferences
         run: tests/scripts/github-action-helper.sh check_ownerreferences
 
diff --git a/pkg/operator/ceph/cluster/osd/deviceset_test.go b/pkg/operator/ceph/cluster/osd/deviceset_test.go
index cbffcbdc4..8f3d0c550 100644
--- a/pkg/operator/ceph/cluster/osd/deviceset_test.go
+++ b/pkg/operator/ceph/cluster/osd/deviceset_test.go
@@ -280,16 +280,20 @@ func TestPrepareDeviceSetsWithCrushParams(t *testing.T) {
 }
 
 func TestPVCName(t *testing.T) {
-	id := deviceSetPVCID("mydeviceset", "a", 0)
-	assert.Equal(t, "mydeviceset-a-0", id)
+	id, err := deviceSetPVCID("mydeviceset-making-the-length-of-pvc-id-greater-than-the-limit-63", "a", 0)
+	assert.Error(t, err)
+	assert.Equal(t, "", id)
 
-	id = deviceSetPVCID("mydeviceset", "a", 10)
+	id, err = deviceSetPVCID("mydeviceset", "a", 10)
+	assert.NoError(t, err)
 	assert.Equal(t, "mydeviceset-a-10", id)
 
-	id = deviceSetPVCID("device-set", "a", 10)
+	id, err = deviceSetPVCID("device-set", "a", 10)
+	assert.NoError(t, err)
 	assert.Equal(t, "device-set-a-10", id)
 
-	id = deviceSetPVCID("device.set.with.dots", "b", 10)
+	id, err = deviceSetPVCID("device.set.with.dots", "b", 10)
+	assert.NoError(t, err)
 	assert.Equal(t, "device-set-with-dots-b-10", id)
 }
 
diff --git a/pkg/operator/ceph/csi/util_test.go b/pkg/operator/ceph/csi/util_test.go
index 190524684..7ea2f09b7 100644
--- a/pkg/operator/ceph/csi/util_test.go
+++ b/pkg/operator/ceph/csi/util_test.go
@@ -120,8 +120,8 @@ func Test_applyVolumeToPodSpec(t *testing.T) {
 		Param:     CSIParam,
 		Namespace: "foo",
 	}
-	// rbdplugin has 11 volumes by default
-	defaultVolumes := 11
+	// rbdplugin has 13 volumes by default
+	defaultVolumes := 13
 	ds, err := templateToDaemonSet(dsName, RBDPluginTemplatePath, tp)
 	assert.Nil(t, err)
 	applyVolumeToPodSpec(config, configKey, &ds.Spec.Template.Spec)
diff --git a/tests/external-cluster/ceph-status-out b/tests/ceph-status-out
similarity index 100%
rename from tests/external-cluster/ceph-status-out
rename to tests/ceph-status-out
diff --git a/tests/external-cluster/external-config.ini b/tests/external-cluster/external-config.ini
deleted file mode 100644
index 20ede38d4..000000000
--- a/tests/external-cluster/external-config.ini
+++ /dev/null
@@ -1,35 +0,0 @@
-[Configurations]
-format =
-verbose =
-ceph-conf =
-keyring =
-run-as-user =
-cluster-name =
-k8s-cluster-name =
-namespace =
-rgw-pool-prefix =
-restricted-auth-permission =
-v2-port-enable =
-output =
-cephfs-filesystem-name =
-cephfs-metadata-pool-name =
-cephfs-data-pool-name =
-rbd-data-pool-name = replicapoolconfig
-alias-rbd-data-pool-name =
-rgw-endpoint =
-rgw-tls-cert-path =
-rgw-skip-tls =
-monitoring-endpoint =
-monitoring-endpoint-port =
-skip-monitoring-endpoint =
-rbd-metadata-ec-pool-name =
-dry-run =
-rados-namespace = radosnamespace2
-subvolume-group =
-rgw-realm-name =
-rgw-zone-name =
-rgw-zonegroup-name =
-topology-pool =
-topology-failure-domain-label =
-topology-failure-domain-values =
-upgrade =