Repository: ceph/ceph-csi
Pull request #2406 (2021-08-12 13:30:40)

rbd: rbd-nbd logging the Ceph-CSI way
<!-- Please take a look at our [Contributing](https://github.com/ceph/ceph-csi/blob/devel/docs/development-guide.md#Code-contribution-workflow)
documentation before submitting a Pull Request!
Thank you for contributing to ceph-csi! -->

# Describe what this PR does #

- Provide host-path for rbd-nbd logging, to enable the logging and persist the logs
- One logfile per device/volume
- Add ability to customize the logDir, default: /var/log/ceph
- Update the meta stash with logDir details
- Use the same to remove logfile on unStage/unmap to be space-efficient
- Provide variable to alter host-path location for ceph clients via helm
- Update rbd-nbd doc about log path details

---

### Summary (behavior with this series)

If you are using the default rbd nodeplugin daemonset and StorageClass templates then `cephLogDir` will be `/var/log/ceph`, this directory will be a host-path and the default log file path will be `/var/log/ceph/rbd-nbd-<volID>.log`. rbd-nbd creates a log file per volume under the `cephLogDir` path on NodeStage(map) and removed the same on the respective NodeUnstage(unmap).

In case if you need a customized log path, you should do the below:

- Edit the daemonset templates to change the `cephLogDir`
  - If you are using helm charts, then you can use key `cephLogDir`

    ```
    helm install --set cephLogDir=/var/log/ceph-csi/my-dir
    ```

  - For standard templates edit [csi-rbdplugin.yaml](https://github.com/ceph/ceph-csi/blob/devel/deploy/rbd/kubernetes/csi-rbdplugin.yaml) to update `hostPath` for `ceph-logdir`, also edit psp [csi-nodeplugin-psp.yaml](https://github.com/ceph/ceph-csi/blob/devel/deploy/rbd/kubernetes/csi-nodeplugin-psp.yaml) to update `pathPrefix` spec entries.
- Update the StorageClass with the customized log directory path
  - Now update rbd StorageClass for `cephLogDir`, for example

    ```
    cephLogDir: "/var/log/prod-A-logs"
    ```

`NOTE`:

- On uninstall make sure to delete `cephLogDir` on host manually to freeup
  some space just in case if there are any uncleaned log files.
- In case if you do not need the rbd-nbd logging to persistent, then just
  update the StorageClass for `cephLogDir` to use a non-persistent path.

---
### Log file naming (with this series)

```
$ ls /var/log/ceph/                                                                                              
rbd-nbd-0001-0024-fed5480a-f00f-417a-a51d-31d8a8144c03-0000000000000003-eba90b33-0156-11ec-a30b-4678a93686c2.log
rbd-nbd-0001-0024-fed5480a-f00f-417a-a51d-31d8a8144c03-0000000000000003-ebdc64a1-0156-11ec-a30b-4678a93686c2.log
rbd-nbd-0001-0024-fed5480a-f00f-417a-a51d-31d8a8144c03-0000000000000003-ec034226-0156-11ec-a30b-4678a93686c2.log
rbd-nbd-0001-0024-fed5480a-f00f-417a-a51d-31d8a8144c03-0000000000000003-ec55a4a7-0156-11ec-a30b-4678a93686c2.log
rbd-nbd-0001-0024-fed5480a-f00f-417a-a51d-31d8a8144c03-0000000000000003-f1111e4e-0155-11ec-a30b-4678a93686c2.log
``` 


#### Also see:

- https://github.com/ceph/ceph-csi/pull/2393
- https://github.com/ceph/ceph-csi/pull/2407


Signed-off-by: Prasanna Kumar Kalever \<prasanna.kalever@redhat.com\>

---

<details>
<summary>Show available bot commands</summary>

These commands are normally not required, but in case of issues, leave any of
the following bot commands in an otherwise empty comment in this PR:

- `/retest ci/centos/<job-name>`: retest the `<job-name>` after unrelated
  failure (please report the failure too!)
- `/retest all`: run this in case the CentOS CI failed to start/report any test
  progress or results

</details>


pkalever: /retest all
pkalever: Most of them failed at deploy, probably something is wrong with this patch, not sure, let me test it once again.
Madhu-1: Do you need to add this Directory to PSP?
 
https://github.com/ceph/ceph-csi/blob/devel/deploy/rbd/kubernetes/csi-nodeplugin-psp.yaml#L39
https://github.com/ceph/ceph-csi/blob/devel/deploy/rbd/kubernetes/csi-provisioner-psp.yaml#L31
https://github.com/ceph/ceph-csi/blob/devel/charts/ceph-csi-rbd/templates/provisioner-psp.yaml#L34
https://github.com/ceph/ceph-csi/blob/devel/charts/ceph-csi-rbd/templates/nodeplugin-psp.yaml#L43
pkalever: > Do you need to add this Directory to PSP?
> 
> https://github.com/ceph/ceph-csi/blob/devel/deploy/rbd/kubernetes/csi-nodeplugin-psp.yaml#L39
> https://github.com/ceph/ceph-csi/blob/devel/deploy/rbd/kubernetes/csi-provisioner-psp.yaml#L31
> https://github.com/ceph/ceph-csi/blob/devel/charts/ceph-csi-rbd/templates/provisioner-psp.yaml#L34
> https://github.com/ceph/ceph-csi/blob/devel/charts/ceph-csi-rbd/templates/nodeplugin-psp.yaml#L43

Oh! probably this is why the tests were failing!?
Thanks @Madhu-1 will fix it.
pkalever: /retest ci/centos/upgrade-tests-cephfs
nixpanic: I have a suggesting to consider for an alternative solution, one that is more container friendly:

- use a shared volume where the `rbd-nbd.log` is stored
- add a side-car to the daemonset that runs `tail -f` on the log from rbd-nbd (retry to run `tail` in case the log does not exist yet)

This makes it possible to get the logs with standard kubernetes calls, and there is no permanent storage on the node needed. It is possible to use this in any environment, not only our e2e.

If there is a log that `rbd map` and similar calls use, there would be the additional benefit that better debugging of Ceph CLI calls becomes possible (just use the same log file).
pkalever: > I have a suggesting to consider for an alternative solution, one that is more container friendly:
> 
>     * use a shared volume where the `rbd-nbd.log` is stored

Do you mean, we don't have to create a host path ourself? Or is it an alternative like the latest changes which gives provision to tweak a custom logfile path from storageclass?

> 
>     * add a side-car to the daemonset that runs `tail -f` on the log from rbd-nbd (retry to run `tail` in case the log does not exist yet)
> 
> 
> This makes it possible to get the logs with standard kubernetes calls, and there is no permanent storage on the node needed. It is possible to use this in any environment, not only our e2e.
> 
> If there is a log that `rbd map` and similar calls use, there would be the additional benefit that better debugging of Ceph CLI calls becomes possible (just use the same log file).

Yes, this should be an alternative to address to mean the K8s standards.
nixpanic: On Wed, Aug 18, 2021 at 05:52:15AM -0700, Prasanna Kumar Kalever wrote:
> > I have a suggesting to consider for an alternative solution, one that is more container friendly:
> > 
> >     * use a shared volume where the `rbd-nbd.log` is stored
> 
> Do you mean, we don't have to create a host path ourself? Or is it an alternative like the latest changes which gives provision to tweak a custom logfile path from storageclass?

Yes, no `hostPath` but a volume with type `emptyDir`. It can be placed
on /var/log/ceph or whatever the default location is for logs.


Madhu-1: > On Wed, Aug 18, 2021 at 05:52:15AM -0700, Prasanna Kumar Kalever wrote: > I have a suggesting to consider for an alternative solution, one that is more container friendly: > > * use a shared volume where the `rbd-nbd.log` is stored Do you mean, we don't have to create a host path ourself? Or is it an alternative like the latest changes which gives provision to tweak a custom logfile path from storageclass?
> Yes, no `hostPath` but a volume with type `emptyDir`. It can be placed on /var/log/ceph or whatever the default location is for logs.

If we use emtpyDir again the logs won't persist for multiple restarts? Can we take the below root for this one?

* Log by default to /var/log/ceph
* Log the rbd-nbd to a different path if it's provided in the storageclass
* template changes to different log path
* update storageclass/documentation for logfile path configuration
* Update documentation for the uninstall

pkalever: > > On Wed, Aug 18, 2021 at 05:52:15AM -0700, Prasanna Kumar Kalever wrote: > I have a suggesting to consider for an alternative solution, one that is more container friendly: > > * use a shared volume where the `rbd-nbd.log` is stored Do you mean, we don't have to create a host path ourself? Or is it an alternative like the latest changes which gives provision to tweak a custom logfile path from storageclass?
> > Yes, no `hostPath` but a volume with type `emptyDir`. It can be placed on /var/log/ceph or whatever the default location is for logs.
> 
> If we use emtpyDir again the logs won't persist for multiple restarts? Can we take the below root for this one?
> 
> * Log by default to /var/log/ceph

Log to **/var/log/ceph/** by default, anyone doesn't need it persisting then just edit the storageclass and change it to a directory that is not bind-mounted.

Example: Maybe
`mapOptions: log-file=/var/lohg/ceph-csi/rbd-nbd.log`

> * Log the rbd-nbd to a different path if it's provided in the storageclass
> * template changes to different log path

That's already handled. I will also add variables for the helm charts case which allows to alter the hostpath and mountpath location.

> * update storageclass/documentation for logfile path configuration
> * Update documentation for the uninstall

OK.


@nixpanic any objections?

Apparently, @Madhu-1 checked rook code and at rook, ceph server logs are well persistent in a hostpath. They also have a `log-collector` sidecar container which logroates the ceph logs at the hostpath directory. Moving forward we can use the same hostpath used by rook and take advantage of logrotation.


pkalever: @Mergifyio rebase
mergify[bot]: **Command `rebase`: success**
> **Branch has been successfully rebased**
> 

pkalever: @Madhu-1 @nixpanic I have updated this PR as per our discussion, please take a look.

Thanks!
pkalever: /retest ci/centos/mini-e2e/k8s-1.21

---

```
! StartHost failed, but will try again: creating host: create: Error creating machine: Error in driver during machine creation: IP not available after waiting: machine minikube didn't return IP after 1 minute
```
pkalever: /retest ci/centos/mini-e2e/k8s-1.21
pkalever: /retest ci/centos/mini-e2e-helm/k8s-1.21

---
```
Aug 23 16:44:11.387: FAIL: failed to write IO, err: command terminated with exit code 2, stdErr: /bin/sh: 1: cannot create /var/lib/www/html/test: Read-only file system

```
pkalever: /retest ci/centos/mini-e2e-helm/k8s-1.21

Patch:
diff --git a/charts/ceph-csi-rbd/README.md b/charts/ceph-csi-rbd/README.md
index 9f26025ed..ef2cc6d51 100644
--- a/charts/ceph-csi-rbd/README.md
+++ b/charts/ceph-csi-rbd/README.md
@@ -138,6 +138,7 @@ charts and their default values.
 | `provisionerSocketFile`                        | The filename of the provisioner socket                                                                                                               | `csi-provisioner.sock`                             |
 | `pluginSocketFile`                             | The filename of the plugin socket                                                                                                                    | `csi.sock`                                         |
 | `kubeletDir`                                   | kubelet working directory                                                                                                                            | `/var/lib/kubelet`                                 |
+| `cephLogDir`                                   | Host path location for ceph client processes logging, ex: rbd-nbd                                                                                    | `/var/log/ceph`                                    |
 | `driverName`                                   | Name of the csi-driver                                                                                                                               | `rbd.csi.ceph.com`                                 |
 | `configMapName`                                | Name of the configmap which contains cluster configuration                                                                                           | `ceph-csi-config`                                  |
 | `externallyManagedConfigmap`                   | Specifies the use of an externally provided configmap                                                                                                | `false`                                            |
diff --git a/charts/ceph-csi-rbd/templates/nodeplugin-daemonset.yaml b/charts/ceph-csi-rbd/templates/nodeplugin-daemonset.yaml
index 09cde8939..003fbf5ec 100644
--- a/charts/ceph-csi-rbd/templates/nodeplugin-daemonset.yaml
+++ b/charts/ceph-csi-rbd/templates/nodeplugin-daemonset.yaml
@@ -124,6 +124,8 @@ spec:
               mountPropagation: "Bidirectional"
             - name: keys-tmp-dir
               mountPath: /tmp/csi/keys
+            - name: ceph-logdir
+              mountPath: /var/log/ceph
           resources:
 {{ toYaml .Values.nodeplugin.plugin.resources | indent 12 }}
 {{- if .Values.nodeplugin.httpMetrics.enabled }}
@@ -169,6 +171,10 @@ spec:
           hostPath:
             path: {{ .Values.kubeletDir }}/pods
             type: DirectoryOrCreate
+        - name: ceph-logdir
+          hostPath:
+            path: {{ .Values.cephLogDir }}
+            type: DirectoryOrCreate
         - name: host-dev
           hostPath:
             path: /dev
diff --git a/charts/ceph-csi-rbd/templates/nodeplugin-psp.yaml b/charts/ceph-csi-rbd/templates/nodeplugin-psp.yaml
index fb9313c11..fc10f134d 100644
--- a/charts/ceph-csi-rbd/templates/nodeplugin-psp.yaml
+++ b/charts/ceph-csi-rbd/templates/nodeplugin-psp.yaml
@@ -42,6 +42,8 @@ spec:
       readOnly: true
     - pathPrefix: '/lib/modules'
       readOnly: true
+    - pathPrefix: '{{ .Values.cephLogDir }}'
+      readOnly: false
     - pathPrefix: '{{ .Values.kubeletDir }}'
       readOnly: false
 {{- end }}
diff --git a/charts/ceph-csi-rbd/values.yaml b/charts/ceph-csi-rbd/values.yaml
index 965baff5a..8a00d96d9 100644
--- a/charts/ceph-csi-rbd/values.yaml
+++ b/charts/ceph-csi-rbd/values.yaml
@@ -382,6 +382,8 @@ provisionerSocketFile: csi-provisioner.sock
 pluginSocketFile: csi.sock
 # kubelet working directory,can be set using `--root-dir` when starting kubelet.
 kubeletDir: /var/lib/kubelet
+# Host path location for ceph client processes logging, ex: rbd-nbd
+cephLogDir: /var/log/ceph
 # Name of the csi-driver
 driverName: rbd.csi.ceph.com
 # Name of the configmap used for state
diff --git a/deploy/rbd/kubernetes/csi-nodeplugin-psp.yaml b/deploy/rbd/kubernetes/csi-nodeplugin-psp.yaml
index 2df3b6cc4..7d735fd86 100644
--- a/deploy/rbd/kubernetes/csi-nodeplugin-psp.yaml
+++ b/deploy/rbd/kubernetes/csi-nodeplugin-psp.yaml
@@ -38,6 +38,8 @@ spec:
       readOnly: true
     - pathPrefix: '/var/lib/kubelet/pods'
       readOnly: false
+    - pathPrefix: '/var/log/ceph'
+      readOnly: false
     - pathPrefix: '/var/lib/kubelet/plugins/rbd.csi.ceph.com'
       readOnly: false
     - pathPrefix: '/var/lib/kubelet/plugins_registry'
diff --git a/deploy/rbd/kubernetes/csi-rbdplugin.yaml b/deploy/rbd/kubernetes/csi-rbdplugin.yaml
index 9a424a7d6..7b7e7aa3f 100644
--- a/deploy/rbd/kubernetes/csi-rbdplugin.yaml
+++ b/deploy/rbd/kubernetes/csi-rbdplugin.yaml
@@ -111,6 +111,8 @@ spec:
               mountPropagation: "Bidirectional"
             - name: keys-tmp-dir
               mountPath: /tmp/csi/keys
+            - name: ceph-logdir
+              mountPath: /var/log/ceph
         - name: liveness-prometheus
           securityContext:
             privileged: true
@@ -146,6 +148,10 @@ spec:
           hostPath:
             path: /var/lib/kubelet/pods
             type: DirectoryOrCreate
+        - name: ceph-logdir
+          hostPath:
+            path: /var/log/ceph
+            type: DirectoryOrCreate
         - name: registration-dir
           hostPath:
             path: /var/lib/kubelet/plugins_registry/
diff --git a/docs/rbd-nbd.md b/docs/rbd-nbd.md
index a24885536..cadf0c400 100644
--- a/docs/rbd-nbd.md
+++ b/docs/rbd-nbd.md
@@ -3,6 +3,7 @@
 - [RBD NBD Mounter](#rbd-nbd-mounter)
   - [Overview](#overview)
   - [Configuration](#configuration)
+    - [Configuring logging path](#configuring-logging-path)
   - [Status](#status)
   - [Support Matrix](#support-matrix)
     - [CSI spec and Kubernetes version compatibility](#csi-spec-and-kubernetes-version-compatibility)
@@ -28,6 +29,41 @@ client-side, which is inside the `csi-rbdplugin` node plugin.
 To use the rbd-nbd mounter for RBD-backed PVs, set `mounter` to `rbd-nbd`
 in the StorageClass.
 
+### Configuring logging path
+
+If you are using the default rbd nodeplugin daemonset and StorageClass
+templates then `cephLogDir` will be `/var/log/ceph`, this directory will be
+a host-path and the default log file path will be
+`/var/log/ceph/rbd-nbd-<volID>.log`. rbd-nbd creates a log file per volume
+under the `cephLogDir` path on NodeStage(map) and removed the same on
+the respective NodeUnstage(unmap).
+
+In case if you need a customized log path, you should do below:
+
+- Edit the daemonset templates to change the `cephLogDir`
+  - If you are using helm charts, then you can use key `cephLogDir`
+
+    ```
+    helm install --set cephLogDir=/var/log/ceph-csi/my-dir
+    ```
+
+  - For standard templates edit [csi-rbdplugin.yaml](../deploy/rbd/kubernetes/csi-rbdplugin.yaml)
+    to update `hostPath` for `ceph-logdir`, also edit psp [csi-nodeplugin-psp.yaml](../deploy/rbd/kubernetes/csi-nodeplugin-psp.yaml)
+    to update `pathPrefix` spec entries.
+- Update the StorageClass with the customized log directory path
+  - Now update rbd StorageClass for `cephLogDir`, for example
+
+    ```
+    cephLogDir: "/var/log/prod-A-logs"
+    ```
+
+`NOTE`:
+
+- On uninstall make sure to delete `cephLogDir` on host manually to freeup
+  some space just in case if there are any uncleaned log files.
+- In case if you do not need the rbd-nbd logging to persistent, then just
+  update the StorageClass for `cephLogDir` to use a non-persistent path.
+
 ## Status
 
 Rbd-nbd support status: **Alpha**
diff --git a/examples/rbd/storageclass.yaml b/examples/rbd/storageclass.yaml
index ced2eaafe..80b421412 100644
--- a/examples/rbd/storageclass.yaml
+++ b/examples/rbd/storageclass.yaml
@@ -69,6 +69,12 @@ parameters:
    # on supported nodes
    # mounter: rbd-nbd
 
+   # (optional) ceph client log location, eg: rbd-nbd
+   # By default host-path /var/log/ceph of node is bind-mounted into
+   # csi-rbdplugin pod at /var/log/ceph mount path. See docs/rbd-nbd.md
+   # for available configuration options.
+   # cephLogDir: /var/log/ceph
+
    # (optional) Prefix to use for naming RBD images.
    # If omitted, defaults to "csi-vol-".
    # volumeNamePrefix: "foo-bar-"
diff --git a/internal/rbd/nodeserver.go b/internal/rbd/nodeserver.go
index 6f6642028..03ddfee6f 100644
--- a/internal/rbd/nodeserver.go
+++ b/internal/rbd/nodeserver.go
@@ -274,6 +274,10 @@ func (ns *NodeServer) NodeStageVolume(
 	volOptions.MapOptions = req.GetVolumeContext()["mapOptions"]
 	volOptions.UnmapOptions = req.GetVolumeContext()["unmapOptions"]
 	volOptions.Mounter = req.GetVolumeContext()["mounter"]
+	volOptions.LogDir = req.GetVolumeContext()["cephLogDir"]
+	if volOptions.LogDir == "" {
+		volOptions.LogDir = defaultLogDir
+	}
 
 	err = volOptions.Connect(cr)
 	if err != nil {
@@ -818,13 +822,17 @@ func (ns *NodeServer) NodeUnstageVolume(
 
 	// Unmapping rbd device
 	imageSpec := imgInfo.String()
-	if err = detachRBDImageOrDeviceSpec(
-		ctx, imageSpec,
-		true,
-		imgInfo.NbdAccess,
-		imgInfo.Encrypted,
-		req.GetVolumeId(),
-		imgInfo.UnmapOptions); err != nil {
+
+	dArgs := detachRBDImageArgs{
+		imageOrDeviceSpec: imageSpec,
+		isImageSpec:       true,
+		isNbd:             imgInfo.NbdAccess,
+		encrypted:         imgInfo.Encrypted,
+		volumeID:          req.GetVolumeId(),
+		unmapOptions:      imgInfo.UnmapOptions,
+		logDir:            imgInfo.LogDir,
+	}
+	if err = detachRBDImageOrDeviceSpec(ctx, dArgs); err != nil {
 		util.ErrorLog(
 			ctx,
 			"error unmapping volume (%s) from staging path (%s): (%v)",
diff --git a/internal/rbd/rbd_attach.go b/internal/rbd/rbd_attach.go
index 81c7487a1..98d16979a 100644
--- a/internal/rbd/rbd_attach.go
+++ b/internal/rbd/rbd_attach.go
@@ -81,6 +81,16 @@ type nbdDeviceInfo struct {
 	Device         string `json:"device"`
 }
 
+type detachRBDImageArgs struct {
+	imageOrDeviceSpec string
+	isImageSpec       bool
+	isNbd             bool
+	encrypted         bool
+	volumeID          string
+	unmapOptions      string
+	logDir            string
+}
+
 // rbdGetDeviceList queries rbd about mapped devices and returns a list of rbdDeviceInfo
 // It will selectively list devices mapped using krbd or nbd as specified by accessType.
 func rbdGetDeviceList(ctx context.Context, accessType string) ([]rbdDeviceInfo, error) {
@@ -296,6 +306,11 @@ func createPath(ctx context.Context, volOpt *rbdVolume, device string, cr *util.
 		util.WarningLog(ctx, "failed to detect if image %q is thick-provisioned: %v", volOpt, err)
 	}
 
+	if isNbd {
+		mapArgs = append(mapArgs, "--log-file",
+			getCephClientLogFileName(volOpt.VolID, volOpt.LogDir, "rbd-nbd"))
+	}
+
 	cli := rbd
 	if device != "" {
 		// TODO: use rbd cli for attach/detach in the future
@@ -317,14 +332,16 @@ func createPath(ctx context.Context, volOpt *rbdVolume, device string, cr *util.
 		util.WarningLog(ctx, "rbd: map error %v, rbd output: %s", err, stderr)
 		// unmap rbd image if connection timeout
 		if strings.Contains(err.Error(), rbdMapConnectionTimeout) {
-			detErr := detachRBDImageOrDeviceSpec(
-				ctx,
-				imagePath,
-				true,
-				isNbd,
-				volOpt.isEncrypted(),
-				volOpt.VolID,
-				volOpt.UnmapOptions)
+			dArgs := detachRBDImageArgs{
+				imageOrDeviceSpec: imagePath,
+				isImageSpec:       true,
+				isNbd:             isNbd,
+				encrypted:         volOpt.isEncrypted(),
+				volumeID:          volOpt.VolID,
+				unmapOptions:      volOpt.UnmapOptions,
+				logDir:            volOpt.LogDir,
+			}
+			detErr := detachRBDImageOrDeviceSpec(ctx, dArgs)
 			if detErr != nil {
 				util.WarningLog(ctx, "rbd: %s unmap error %v", imagePath, detErr)
 			}
@@ -367,22 +384,29 @@ func detachRBDDevice(ctx context.Context, devicePath, volumeID, unmapOptions str
 		nbdType = true
 	}
 
-	return detachRBDImageOrDeviceSpec(ctx, devicePath, false, nbdType, encrypted, volumeID, unmapOptions)
+	dArgs := detachRBDImageArgs{
+		imageOrDeviceSpec: devicePath,
+		isImageSpec:       false,
+		isNbd:             nbdType,
+		encrypted:         encrypted,
+		volumeID:          volumeID,
+		unmapOptions:      unmapOptions,
+	}
+
+	return detachRBDImageOrDeviceSpec(ctx, dArgs)
 }
 
 // detachRBDImageOrDeviceSpec detaches an rbd imageSpec or devicePath, with additional checking
 // when imageSpec is used to decide if image is already unmapped.
 func detachRBDImageOrDeviceSpec(
 	ctx context.Context,
-	imageOrDeviceSpec string,
-	isImageSpec, isNbd, encrypted bool,
-	volumeID, unmapOptions string) error {
-	if encrypted {
-		mapperFile, mapperPath := util.VolumeMapper(volumeID)
+	dArgs detachRBDImageArgs) error {
+	if dArgs.encrypted {
+		mapperFile, mapperPath := util.VolumeMapper(dArgs.volumeID)
 		mappedDevice, mapper, err := util.DeviceEncryptionStatus(ctx, mapperPath)
 		if err != nil {
 			util.ErrorLog(ctx, "error determining LUKS device on %s, %s: %s",
-				mapperPath, imageOrDeviceSpec, err)
+				mapperPath, dArgs.imageOrDeviceSpec, err)
 
 			return err
 		}
@@ -391,31 +415,38 @@ func detachRBDImageOrDeviceSpec(
 			err = util.CloseEncryptedVolume(ctx, mapperFile)
 			if err != nil {
 				util.ErrorLog(ctx, "error closing LUKS device on %s, %s: %s",
-					mapperPath, imageOrDeviceSpec, err)
+					mapperPath, dArgs.imageOrDeviceSpec, err)
 
 				return err
 			}
-			imageOrDeviceSpec = mappedDevice
+			dArgs.imageOrDeviceSpec = mappedDevice
 		}
 	}
 
-	unmapArgs := []string{"unmap", imageOrDeviceSpec}
-	unmapArgs = appendDeviceTypeAndOptions(unmapArgs, isNbd, false, unmapOptions)
+	unmapArgs := []string{"unmap", dArgs.imageOrDeviceSpec}
+	unmapArgs = appendDeviceTypeAndOptions(unmapArgs, dArgs.isNbd, false, dArgs.unmapOptions)
 
 	_, stderr, err := util.ExecCommand(ctx, rbd, unmapArgs...)
 	if err != nil {
 		// Messages for krbd and nbd differ, hence checking either of them for missing mapping
 		// This is not applicable when a device path is passed in
-		if isImageSpec &&
-			(strings.Contains(stderr, fmt.Sprintf(rbdUnmapCmdkRbdMissingMap, imageOrDeviceSpec)) ||
-				strings.Contains(stderr, fmt.Sprintf(rbdUnmapCmdNbdMissingMap, imageOrDeviceSpec))) {
+		if dArgs.isImageSpec &&
+			(strings.Contains(stderr, fmt.Sprintf(rbdUnmapCmdkRbdMissingMap, dArgs.imageOrDeviceSpec)) ||
+				strings.Contains(stderr, fmt.Sprintf(rbdUnmapCmdNbdMissingMap, dArgs.imageOrDeviceSpec))) {
 			// Devices found not to be mapped are treated as a successful detach
-			util.TraceLog(ctx, "image or device spec (%s) not mapped", imageOrDeviceSpec)
+			util.TraceLog(ctx, "image or device spec (%s) not mapped", dArgs.imageOrDeviceSpec)
 
 			return nil
 		}
 
-		return fmt.Errorf("rbd: unmap for spec (%s) failed (%w): (%s)", imageOrDeviceSpec, err, stderr)
+		return fmt.Errorf("rbd: unmap for spec (%s) failed (%w): (%s)", dArgs.imageOrDeviceSpec, err, stderr)
+	}
+	if dArgs.isNbd && dArgs.logDir != "" {
+		logFile := getCephClientLogFileName(dArgs.volumeID, dArgs.logDir, "rbd-nbd")
+		if err = os.Remove(logFile); err != nil {
+			util.WarningLog(ctx, "failed to remove logfile: %s, error: %v",
+				logFile, err)
+		}
 	}
 
 	return nil
diff --git a/internal/rbd/rbd_util.go b/internal/rbd/rbd_util.go
index 8a6a24352..ad5727629 100644
--- a/internal/rbd/rbd_util.go
+++ b/internal/rbd/rbd_util.go
@@ -50,6 +50,7 @@ const (
 	rbdImageWatcherSteps     = 10
 	rbdDefaultMounter        = "rbd"
 	rbdNbdMounter            = "rbd-nbd"
+	defaultLogDir            = "/var/log/ceph"
 
 	// Output strings returned during invocation of "ceph rbd task add remove <imagespec>" when
 	// command is not supported by ceph manager. Used to check errors and recover when the command
@@ -136,6 +137,7 @@ type rbdVolume struct {
 	ReservedID         string
 	MapOptions         string
 	UnmapOptions       string
+	LogDir             string
 	VolName            string `json:"volName"`
 	MonValueFromSecret string `json:"monValueFromSecret"`
 	VolSize            int64  `json:"volSize"`
@@ -1523,6 +1525,7 @@ type rbdImageMetadataStash struct {
 	NbdAccess      bool   `json:"accessType"`
 	Encrypted      bool   `json:"encrypted"`
 	DevicePath     string `json:"device"` // holds NBD device path for now
+	LogDir         string `json:"logDir"` // holds the client log path
 }
 
 // file name in which image metadata is stashed.
@@ -1553,6 +1556,7 @@ func stashRBDImageMetadata(volOptions *rbdVolume, metaDataPath string) error {
 	imgMeta.NbdAccess = false
 	if volOptions.Mounter == rbdTonbd && hasNBD {
 		imgMeta.NbdAccess = true
+		imgMeta.LogDir = volOptions.LogDir
 	}
 
 	encodedBytes, err := json.Marshal(imgMeta)
@@ -2002,3 +2006,16 @@ func (ri *rbdImage) addSnapshotScheduling(
 
 	return nil
 }
+
+// getCephClientLogFileName compiles the complete log file path based on inputs.
+func getCephClientLogFileName(id, logDir, prefix string) string {
+	if prefix == "" {
+		prefix = "ceph"
+	}
+
+	if logDir == "" {
+		logDir = defaultLogDir
+	}
+
+	return fmt.Sprintf("%s/%s-%s.log", logDir, prefix, id)
+}


Test patch:
diff --git a/internal/rbd/rbd_util_test.go b/internal/rbd/rbd_util_test.go
index 1dfa35d7d..2a44682db 100644
--- a/internal/rbd/rbd_util_test.go
+++ b/internal/rbd/rbd_util_test.go
@@ -189,3 +189,74 @@ func TestGetMappedID(t *testing.T) {
 		})
 	}
 }
+
+func TestGetCephClientLogFileName(t *testing.T) {
+	t.Parallel()
+	type args struct {
+		id     string
+		logDir string
+		prefix string
+	}
+	volID := "0001-0024-fed5480a-f00f-417a-a51d-31d8a8144c03-0000000000000003-eba90b33-0156-11ec-a30b-4678a93686c2"
+	tests := []struct {
+		name     string
+		args     args
+		expected string
+	}{
+		{
+			name: "test for empty id",
+			args: args{
+				id:     "",
+				logDir: "/var/log/ceph-csi",
+				prefix: "rbd-nbd",
+			},
+			expected: "/var/log/ceph-csi/rbd-nbd-.log",
+		},
+		{
+			name: "test for empty logDir",
+			args: args{
+				id:     volID,
+				logDir: "",
+				prefix: "rbd-nbd",
+			},
+			expected: "/var/log/ceph/rbd-nbd-" + volID + ".log",
+		},
+		{
+			name: "test for empty prefix",
+			args: args{
+				id:     volID,
+				logDir: "/var/log/ceph-csi",
+				prefix: "",
+			},
+			expected: "/var/log/ceph-csi/ceph-" + volID + ".log",
+		},
+		{
+			name: "test for all unavailable args",
+			args: args{
+				id:     "",
+				logDir: "",
+				prefix: "",
+			},
+			expected: "/var/log/ceph/ceph-.log",
+		},
+		{
+			name: "test for all available args",
+			args: args{
+				id:     volID,
+				logDir: "/var/log/ceph-csi",
+				prefix: "rbd-nbd",
+			},
+			expected: "/var/log/ceph-csi/rbd-nbd-" + volID + ".log",
+		},
+	}
+	for _, tt := range tests {
+		tt := tt
+		t.Run(tt.name, func(t *testing.T) {
+			t.Parallel()
+			val := getCephClientLogFileName(tt.args.id, tt.args.logDir, tt.args.prefix)
+			if val != tt.expected {
+				t.Errorf("getCephClientLogFileName() got = %v, expected %v", val, tt.expected)
+			}
+		})
+	}
+}