Repository: ceph/ceph-csi (go)

Issue:
RBD DR: Returning false success  response during ResyncVolume
```
2022-08-04T18:09:11.765341626Z I0804 18:09:11.765320       1 utils.go:191] ID: 563 Req-ID: 0001-0011-openshift-storage-0000000000000001-f47e367f-141a-11ed-8b38-0a580a83000a GRPC call: /replication.Controller/ResyncVolume
2022-08-04T18:09:11.765430869Z I0804 18:09:11.765414       1 utils.go:193] ID: 563 Req-ID: 0001-0011-openshift-storage-0000000000000001-f47e367f-141a-11ed-8b38-0a580a83000a GRPC request: {"force":true,"parameters":{"mirroringMode":"snapshot","schedulingInterval":"5m"},"secrets":"***stripped***","volume_id":"0001-0011-openshift-storage-0000000000000001-f47e367f-141a-11ed-8b38-0a580a83000a"}
2022-08-04T18:09:11.767060885Z I0804 18:09:11.767023       1 omap.go:87] ID: 563 Req-ID: 0001-0011-openshift-storage-0000000000000001-f47e367f-141a-11ed-8b38-0a580a83000a got omap values: (pool="ocs-storagecluster-cephblockpool", namespace="", name="csi.volume.f47e367f-141a-11ed-8b38-0a580a83000a"): map[csi.imageid:1714da6aa3421 csi.imagename:csi-vol-f47e367f-141a-11ed-8b38-0a580a83000a csi.volname:pvc-796a60a7-2160-4c99-99e4-6205274b711b csi.volume.owner:workload-2]
2022-08-04T18:09:11.827970147Z I0804 18:09:11.827897       1 replicationcontrollerserver.go:781] ID: 563 Req-ID: 0001-0011-openshift-storage-0000000000000001-f47e367f-141a-11ed-8b38-0a580a83000a local status: daemon up=false, image mirroring state="unknown", description="status not found" and lastUpdate=1970-01-01 00:00:00 +0000 UTC
2022-08-04T18:09:11.828042369Z I0804 18:09:11.828031       1 utils.go:202] ID: 563 Req-ID: 0001-0011-openshift-storage-0000000000000001-f47e367f-141a-11ed-8b38-0a580a83000a GRPC response: {}
```

```
2022-08-04T18:09:17.725369648Z I0804 18:09:17.725311       1 utils.go:191] ID: 572 Req-ID: 0001-0011-openshift-storage-0000000000000001-f47e367f-141a-11ed-8b38-0a580a83000a GRPC call: /replication.Controller/DisableVolumeReplication
2022-08-04T18:09:17.725398830Z I0804 18:09:17.725392       1 utils.go:193] ID: 572 Req-ID: 0001-0011-openshift-storage-0000000000000001-f47e367f-141a-11ed-8b38-0a580a83000a GRPC request: {"parameters":{"mirroringMode":"snapshot","schedulingInterval":"5m"},"secrets":"***stripped***","volume_id":"0001-0011-openshift-storage-0000000000000001-f47e367f-141a-11ed-8b38-0a580a83000a"}
2022-08-04T18:09:17.727219222Z I0804 18:09:17.727179       1 omap.go:87] ID: 572 Req-ID: 0001-0011-openshift-storage-0000000000000001-f47e367f-141a-11ed-8b38-0a580a83000a got omap values: (pool="ocs-storagecluster-cephblockpool", namespace="", name="csi.volume.f47e367f-141a-11ed-8b38-0a580a83000a"): map[csi.imageid:1714da6aa3421 csi.imagename:csi-vol-f47e367f-141a-11ed-8b38-0a580a83000a csi.volname:pvc-796a60a7-2160-4c99-99e4-6205274b711b csi.volume.owner:workload-2]
2022-08-04T18:09:17.744785533Z W0804 18:09:17.744709       1 replicationcontrollerserver.go:99] ID: 572 Req-ID: 0001-0011-openshift-storage-0000000000000001-f47e367f-141a-11ed-8b38-0a580a83000a force is not set in parameters, setting to default (false)
2022-08-04T18:09:17.777460766Z E0804 18:09:17.777381       1 utils.go:200] ID: 572 Req-ID: 0001-0011-openshift-storage-0000000000000001-f47e367f-141a-11ed-8b38-0a580a83000a GRPC error: rpc error: code = InvalidArgument desc = secondary image status is up=false and state=unknown

2022-08-04T19:04:30.946578430Z I0804 19:04:30.946527       1 utils.go:191] ID: 745 Req-ID: 0001-0011-openshift-storage-0000000000000001-f47e367f-141a-11ed-8b38-0a580a83000a GRPC call: /replication.Controller/DisableVolumeReplication
2022-08-04T19:04:30.946640637Z I0804 19:04:30.946625       1 utils.go:193] ID: 745 Req-ID: 0001-0011-openshift-storage-0000000000000001-f47e367f-141a-11ed-8b38-0a580a83000a GRPC request: {"parameters":{"mirroringMode":"snapshot","schedulingInterval":"5m"},"secrets":"***stripped***","volume_id":"0001-0011-openshift-storage-0000000000000001-f47e367f-141a-11ed-8b38-0a580a83000a"}
2022-08-04T19:04:30.948866897Z I0804 19:04:30.948787       1 omap.go:87] ID: 745 Req-ID: 0001-0011-openshift-storage-0000000000000001-f47e367f-141a-11ed-8b38-0a580a83000a got omap values: (pool="ocs-storagecluster-cephblockpool", namespace="", name="csi.volume.f47e367f-141a-11ed-8b38-0a580a83000a"): map[csi.imageid:1714da6aa3421 csi.imagename:csi-vol-f47e367f-141a-11ed-8b38-0a580a83000a csi.volname:pvc-796a60a7-2160-4c99-99e4-6205274b711b csi.volume.owner:workload-2]
2022-08-04T19:04:30.968659306Z W0804 19:04:30.968598       1 replicationcontrollerserver.go:99] ID: 745 Req-ID: 0001-0011-openshift-storage-0000000000000001-f47e367f-141a-11ed-8b38-0a580a83000a force is not set in parameters, setting to default (false)
2022-08-04T19:04:31.014579039Z E0804 19:04:31.014400       1 utils.go:200] ID: 745 Req-ID: 0001-0011-openshift-storage-0000000000000001-f47e367f-141a-11ed-8b38-0a580a83000a GRPC error: rpc error: code = InvalidArgument desc = secondary image status is up=true and state=error
```

If the rbd mirror daemon is not running on the cluster, the rbd image mirror status will not get updated as expected. As we dont have a check for the rbd mirror daemon, we are returning a resync success response and Ready=false. Due to this false response, Ramen is assuming the resync started and deleted the volume replication CR. Because of this, the cluster goes into a bad state and needs manual intervention.

If the mirror daemon is not running we cannot do anything, we  need to add  a check for rbd mirror daemon to avoid false response.

Pull request:
rbd: consider mirror deamon state for ResyncVolume
During ResyncVolume, we check if the image is in an error state, and we resync. After resync, the image will move to either the `Error` or the `Resyncing` state. And if the image is in the above two conditions, we will return a successful response and Ready=false so that the consumer can wait until the volume is ready to use. If the image is in any other state we return an error message to indicate the syncing is not going on. The whole resync and image state change depends on the rbd mirror daemon. If the mirror daemon is not running, the image can be in Resyncing or Unknown state. The Ramen marks the volume replication as secondary, and once the resync starts, it will delete the volume replication CR as a cleanup process.

As we dont have a check for the rbd mirror daemon, we are returning a resync success response and Ready=false. Due to this false response, Ramen is assuming the resync started and deleted the volume replication CR. Because of this, the cluster goes into a bad state and needs manual intervention.

fixes #3289

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>



Patch:
diff --git a/internal/rbd/replicationcontrollerserver.go b/internal/rbd/replicationcontrollerserver.go
index e93887686..7393c2fdd 100644
--- a/internal/rbd/replicationcontrollerserver.go
+++ b/internal/rbd/replicationcontrollerserver.go
@@ -856,12 +856,13 @@ func checkVolumeResyncStatus(localStatus librbd.SiteMirrorImageStatus) error {
 
 	// If the state is Replaying means the resync is going on.
 	// Once the volume on remote cluster is demoted and resync
-	// is completed the image state will be moved to UNKNOWN .
-	if localStatus.State != librbd.MirrorImageStatusStateReplaying &&
-		localStatus.State != librbd.MirrorImageStatusStateUnknown {
+	// is completed the image state will be moved to UNKNOWN.
+	// RBD mirror daemon should be always running on the primary cluster.
+	if !localStatus.Up || (localStatus.State != librbd.MirrorImageStatusStateReplaying &&
+		localStatus.State != librbd.MirrorImageStatusStateUnknown) {
 		return fmt.Errorf(
-			"not resyncing. image is in %q state",
-			localStatus.State)
+			"not resyncing. Local status: daemon up=%t image is in %q state",
+			localStatus.Up, localStatus.State)
 	}
 
 	return nil


Test patch:
diff --git a/internal/rbd/replicationcontrollerserver_test.go b/internal/rbd/replicationcontrollerserver_test.go
index 401292587..e10d35249 100644
--- a/internal/rbd/replicationcontrollerserver_test.go
+++ b/internal/rbd/replicationcontrollerserver_test.go
@@ -212,10 +212,19 @@ func TestCheckVolumeResyncStatus(t *testing.T) {
 		args    librbd.SiteMirrorImageStatus
 		wantErr bool
 	}{
+		{
+			name: "test when rbd mirror daemon is not running",
+			args: librbd.SiteMirrorImageStatus{
+				State: librbd.MirrorImageStatusStateUnknown,
+				Up:    false,
+			},
+			wantErr: true,
+		},
 		{
 			name: "test for unknown state",
 			args: librbd.SiteMirrorImageStatus{
 				State: librbd.MirrorImageStatusStateUnknown,
+				Up:    true,
 			},
 			wantErr: false,
 		},
@@ -223,6 +232,7 @@ func TestCheckVolumeResyncStatus(t *testing.T) {
 			name: "test for error state",
 			args: librbd.SiteMirrorImageStatus{
 				State: librbd.MirrorImageStatusStateError,
+				Up:    true,
 			},
 			wantErr: true,
 		},
@@ -230,6 +240,7 @@ func TestCheckVolumeResyncStatus(t *testing.T) {
 			name: "test for syncing state",
 			args: librbd.SiteMirrorImageStatus{
 				State: librbd.MirrorImageStatusStateSyncing,
+				Up:    true,
 			},
 			wantErr: true,
 		},
@@ -237,6 +248,7 @@ func TestCheckVolumeResyncStatus(t *testing.T) {
 			name: "test for starting_replay state",
 			args: librbd.SiteMirrorImageStatus{
 				State: librbd.MirrorImageStatusStateStartingReplay,
+				Up:    true,
 			},
 			wantErr: true,
 		},
@@ -244,6 +256,7 @@ func TestCheckVolumeResyncStatus(t *testing.T) {
 			name: "test for replaying state",
 			args: librbd.SiteMirrorImageStatus{
 				State: librbd.MirrorImageStatusStateReplaying,
+				Up:    true,
 			},
 			wantErr: false,
 		},
@@ -251,6 +264,7 @@ func TestCheckVolumeResyncStatus(t *testing.T) {
 			name: "test for stopping_replay state",
 			args: librbd.SiteMirrorImageStatus{
 				State: librbd.MirrorImageStatusStateStoppingReplay,
+				Up:    true,
 			},
 			wantErr: true,
 		},
@@ -258,6 +272,7 @@ func TestCheckVolumeResyncStatus(t *testing.T) {
 			name: "test for stopped state",
 			args: librbd.SiteMirrorImageStatus{
 				State: librbd.MirrorImageStatusStateStopped,
+				Up:    true,
 			},
 			wantErr: true,
 		},
@@ -265,6 +280,7 @@ func TestCheckVolumeResyncStatus(t *testing.T) {
 			name: "test for invalid state",
 			args: librbd.SiteMirrorImageStatus{
 				State: librbd.MirrorImageStatusState(100),
+				Up:    true,
 			},
 			wantErr: true,
 		},