Repository: rook/rook
Pull request #8425 (2021-07-29 05:43:34)

ceph: update cephcsi to v3.4.0
<!-- Please take a look at our [Contributing](https://rook.io/docs/rook/master/development-flow.html)
documentation before submitting a Pull Request!
Thank you for contributing to Rook! -->

**Description of your changes:**

This PR updates the required RBAC, templates, CSI image version, and examples to support the cephcsi v3.4.0 release.

Release notes  available at https://github.com/ceph/ceph-csi/releases/tag/v3.4.0

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>

**Which issue is resolved by this Pull Request:**
Resolves #

**Checklist:**

- [ ] **Commit Message Formatting**: Commit titles and messages follow guidelines in the [developer guide](https://rook.io/docs/rook/master/development-flow.html#commit-structure).
- [ ] **Skip Tests for Docs**: Add the flag for skipping the build if this is only a documentation change. See [here](https://github.com/rook/rook/blob/master/INSTALL.md#skip-ci) for the flag.
- [ ] **Skip Unrelated Tests**: Add a flag to run tests for a specific storage provider. See [test options](https://github.com/rook/rook/blob/master/INSTALL.md#test-storage-provider).
- [ ] Reviewed the developer guide on [Submitting a Pull Request](https://rook.io/docs/rook/master/development-flow.html#submitting-a-pull-request)
- [ ] Documentation has been updated, if necessary.
- [ ] Unit tests have been added, if necessary.
- [ ] Integration tests have been added, if necessary.
- [ ] Pending release notes updated with breaking and/or notable changes, if necessary.
- [ ] Upgrade from previous release is tested and upgrade user guide is updated, if necessary.
- [ ] Code generation (`make codegen`) has been run to update object specifications, if necessary.



Madhu-1: cc @humblec @nixpanic @pkalever please check if am missing anything
Madhu-1: Added DNM till cephcsi 3.4.0 release is out.
pkalever: LGTM, Thanks @Madhu-1 :+1: 
Madhu-1: Removing DNM as cephcsi v3.4.0 image is available.
Madhu-1: > Please edit the Rook pending release note :)

Done! PTAL
mergify[bot]: This pull request has merge conflicts that must be resolved before it can be merged. @Madhu-1 please rebase it. https://rook.io/docs/rook/master/development-flow.html#updating-your-fork

Patch:
diff --git a/Documentation/ceph-upgrade.md b/Documentation/ceph-upgrade.md
index d1fa68084..e6607ed65 100644
--- a/Documentation/ceph-upgrade.md
+++ b/Documentation/ceph-upgrade.md
@@ -463,7 +463,7 @@ kubectl -n $ROOK_OPERATOR_NAMESPACE edit configmap rook-ceph-operator-config
 The default upstream images are included below, which you can change to your desired images.
 
 ```yaml
-ROOK_CSI_CEPH_IMAGE: "quay.io/cephcsi/cephcsi:v3.3.1"
+ROOK_CSI_CEPH_IMAGE: "quay.io/cephcsi/cephcsi:v3.4.0"
 ROOK_CSI_REGISTRAR_IMAGE: "k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0"
 ROOK_CSI_PROVISIONER_IMAGE: "k8s.gcr.io/sig-storage/csi-provisioner:v2.2.2"
 ROOK_CSI_ATTACHER_IMAGE: "k8s.gcr.io/sig-storage/csi-attacher:v3.2.1"
@@ -493,6 +493,6 @@ k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0
 k8s.gcr.io/sig-storage/csi-provisioner:v2.2.2
 k8s.gcr.io/sig-storage/csi-resizer:v1.2.0
 k8s.gcr.io/sig-storage/csi-snapshotter:v4.1.1
-quay.io/cephcsi/cephcsi:v3.3.1
+quay.io/cephcsi/cephcsi:v3.4.0
 quay.io/csiaddons/volumereplication-operator:v0.1.0
 ```
diff --git a/Documentation/helm-operator.md b/Documentation/helm-operator.md
index 2812a1b63..b512430d0 100644
--- a/Documentation/helm-operator.md
+++ b/Documentation/helm-operator.md
@@ -132,7 +132,7 @@ The following tables lists the configurable parameters of the rook-operator char
 | `csi.rbdLivenessMetricsPort`        | Ceph CSI RBD driver metrics port.                                                                                           | `8080`                                                    |
 | `csi.forceCephFSKernelClient`       | Enable Ceph Kernel clients on kernel < 4.17 which support quotas for Cephfs.                                                | `true`                                                    |
 | `csi.kubeletDirPath`                | Kubelet root directory path (if the Kubelet uses a different path for the `--root-dir` flag)                                | `/var/lib/kubelet`                                        |
-| `csi.cephcsi.image`                 | Ceph CSI image.                                                                                                             | `quay.io/cephcsi/cephcsi:v3.3.1`                          |
+| `csi.cephcsi.image`                 | Ceph CSI image.                                                                                                             | `quay.io/cephcsi/cephcsi:v3.4.0`                          |
 | `csi.rbdPluginUpdateStrategy`       | CSI Rbd plugin daemonset update strategy, supported values are OnDelete and RollingUpdate.                                  | `OnDelete`                                                |
 | `csi.cephFSPluginUpdateStrategy`    | CSI CephFS plugin daemonset update strategy, supported values are OnDelete and RollingUpdate.                               | `OnDelete`                                                |
 | `csi.registrar.image`               | Kubernetes CSI registrar image.                                                                                             | `k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0` |
diff --git a/PendingReleaseNotes.md b/PendingReleaseNotes.md
index 0069361af..ec2aabaeb 100644
--- a/PendingReleaseNotes.md
+++ b/PendingReleaseNotes.md
@@ -41,6 +41,7 @@ So the CephCLuster spec field `image` must be updated to point to quay, like `im
 - Stretch clusters are considered stable
   - Ceph v16.2.5 or greater is required for stretch clusters
 - The use of peer secret names in CephRBDMirror is deprecated. Please use CephBlockPool CR to configure peer secret names and import peers. Checkout the `mirroring` section in the CephBlockPool [spec](Documentation/ceph-pool-crd.md#spec) for more details.
+- Update Ceph CSI to `v3.4.0` for more details read the [official release note](https://github.com/ceph/ceph-csi/releases/tag/v3.4.0)
 
 ### Cassandra
 
diff --git a/cluster/charts/rook-ceph/templates/clusterrole.yaml b/cluster/charts/rook-ceph/templates/clusterrole.yaml
index dc62d8fea..2218ec33d 100644
--- a/cluster/charts/rook-ceph/templates/clusterrole.yaml
+++ b/cluster/charts/rook-ceph/templates/clusterrole.yaml
@@ -378,6 +378,9 @@ rules:
   - apiGroups: [""]
     resources: ["configmaps"]
     verbs: ["get", "list"]
+  - apiGroups: [""]
+    resources: ["serviceaccounts"]
+    verbs: ["get"]
 ---
 kind: ClusterRole
 apiVersion: rbac.authorization.k8s.io/v1
@@ -444,6 +447,9 @@ rules:
   - apiGroups: ["replication.storage.openshift.io"]
     resources: ["volumereplicationclasses/status"]
     verbs: ["get"]
+  - apiGroups: [""]
+    resources: ["serviceaccounts"]
+    verbs: ["get"]
 {{- end }}
 {{- if .Values.pspEnable }}
 ---
diff --git a/cluster/charts/rook-ceph/values.yaml b/cluster/charts/rook-ceph/values.yaml
index ef4221726..c7356f4f2 100644
--- a/cluster/charts/rook-ceph/values.yaml
+++ b/cluster/charts/rook-ceph/values.yaml
@@ -270,7 +270,7 @@ csi:
   #rbdLivenessMetricsPort: 9080
   #kubeletDirPath: /var/lib/kubelet
   #cephcsi:
-    #image: quay.io/cephcsi/cephcsi:v3.3.1
+    #image: quay.io/cephcsi/cephcsi:v3.4.0
   #registrar:
     #image: k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0
   #provisioner:
diff --git a/cluster/examples/kubernetes/ceph/common.yaml b/cluster/examples/kubernetes/ceph/common.yaml
index 8a3ad35d3..d174d3f0d 100644
--- a/cluster/examples/kubernetes/ceph/common.yaml
+++ b/cluster/examples/kubernetes/ceph/common.yaml
@@ -1072,6 +1072,9 @@ rules:
   - apiGroups: [""]
     resources: ["configmaps"]
     verbs: ["get", "list"]
+  - apiGroups: [""]
+    resources: ["serviceaccounts"]
+    verbs: ["get"]
 ---
 kind: ClusterRole
 apiVersion: rbac.authorization.k8s.io/v1
@@ -1138,6 +1141,9 @@ rules:
   - apiGroups: ["replication.storage.openshift.io"]
     resources: ["volumereplicationclasses/status"]
     verbs: ["get"]
+  - apiGroups: [""]
+    resources: ["serviceaccounts"]
+    verbs: ["get"]
 # OLM: END CSI RBD CLUSTER ROLE
 # OLM: BEGIN CSI RBD CLUSTER ROLEBINDING
 ---
diff --git a/cluster/examples/kubernetes/ceph/csi/rbd/storageclass-ec.yaml b/cluster/examples/kubernetes/ceph/csi/rbd/storageclass-ec.yaml
index f49f2fa89..c62507ffc 100644
--- a/cluster/examples/kubernetes/ceph/csi/rbd/storageclass-ec.yaml
+++ b/cluster/examples/kubernetes/ceph/csi/rbd/storageclass-ec.yaml
@@ -77,9 +77,9 @@ parameters:
   # will set default as `ext4`.
   csi.storage.k8s.io/fstype: ext4
 # uncomment the following to use rbd-nbd as mounter on supported nodes
-# **IMPORTANT**: If you are using rbd-nbd as the mounter, during upgrade you will be hit a ceph-csi
-# issue that causes the mount to be disconnected. You will need to follow special upgrade steps
-# to restart your application pods. Therefore, this option is not recommended.
+# **IMPORTANT**: CephCSI v3.4.0 onwards a volume healer functionality is added to reattach
+# the PVC to application pod if nodeplugin pod restart.
+# Its still in Alpha support. Therefore, this option is not recommended for production use.
 #mounter: rbd-nbd
 allowVolumeExpansion: true
 reclaimPolicy: Delete
diff --git a/cluster/examples/kubernetes/ceph/csi/rbd/storageclass.yaml b/cluster/examples/kubernetes/ceph/csi/rbd/storageclass.yaml
index 8077139fc..98ef451f7 100644
--- a/cluster/examples/kubernetes/ceph/csi/rbd/storageclass.yaml
+++ b/cluster/examples/kubernetes/ceph/csi/rbd/storageclass.yaml
@@ -66,9 +66,9 @@ parameters:
   # in hyperconverged settings where the volume is mounted on the same node as the osds.
   csi.storage.k8s.io/fstype: ext4
 # uncomment the following to use rbd-nbd as mounter on supported nodes
-# **IMPORTANT**: If you are using rbd-nbd as the mounter, during upgrade you will be hit a ceph-csi
-# issue that causes the mount to be disconnected. You will need to follow special upgrade steps
-# to restart your application pods. Therefore, this option is not recommended.
+# **IMPORTANT**: CephCSI v3.4.0 onwards a volume healer functionality is added to reattach
+# the PVC to application pod if nodeplugin pod restart.
+# Its still in Alpha support. Therefore, this option is not recommended for production use.
 #mounter: rbd-nbd
 allowVolumeExpansion: true
 reclaimPolicy: Delete
diff --git a/cluster/examples/kubernetes/ceph/csi/template/rbd/csi-rbdplugin.yaml b/cluster/examples/kubernetes/ceph/csi/template/rbd/csi-rbdplugin.yaml
index 1d1c9bfe1..5e83ac894 100644
--- a/cluster/examples/kubernetes/ceph/csi/template/rbd/csi-rbdplugin.yaml
+++ b/cluster/examples/kubernetes/ceph/csi/template/rbd/csi-rbdplugin.yaml
@@ -67,6 +67,7 @@ spec:
             - "--metricsport={{ .RBDGRPCMetricsPort }}"
             - "--metricspath=/metrics"
             - "--enablegrpcmetrics={{ .EnableCSIGRPCMetrics }}"
+            - "--stagingpath={{ .KubeletDirPath }}/plugins/kubernetes.io/csi/pv/"
           env:
             - name: POD_IP
               valueFrom:
diff --git a/cluster/examples/kubernetes/ceph/operator-openshift.yaml b/cluster/examples/kubernetes/ceph/operator-openshift.yaml
index 4b06c3873..1171f7c95 100644
--- a/cluster/examples/kubernetes/ceph/operator-openshift.yaml
+++ b/cluster/examples/kubernetes/ceph/operator-openshift.yaml
@@ -148,7 +148,7 @@ data:
   # The default version of CSI supported by Rook will be started. To change the version
   # of the CSI driver to something other than what is officially supported, change
   # these images to the desired release of the CSI driver.
-  # ROOK_CSI_CEPH_IMAGE: "quay.io/cephcsi/cephcsi:v3.3.1"
+  # ROOK_CSI_CEPH_IMAGE: "quay.io/cephcsi/cephcsi:v3.4.0"
   # ROOK_CSI_REGISTRAR_IMAGE: "k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0"
   # ROOK_CSI_RESIZER_IMAGE: "k8s.gcr.io/sig-storage/csi-resizer:v1.2.0"
   # ROOK_CSI_PROVISIONER_IMAGE: "k8s.gcr.io/sig-storage/csi-provisioner:v2.2.2"
diff --git a/cluster/examples/kubernetes/ceph/operator.yaml b/cluster/examples/kubernetes/ceph/operator.yaml
index 9ddc8ffec..02cbd7b07 100644
--- a/cluster/examples/kubernetes/ceph/operator.yaml
+++ b/cluster/examples/kubernetes/ceph/operator.yaml
@@ -72,7 +72,7 @@ data:
   # The default version of CSI supported by Rook will be started. To change the version
   # of the CSI driver to something other than what is officially supported, change
   # these images to the desired release of the CSI driver.
-  # ROOK_CSI_CEPH_IMAGE: "quay.io/cephcsi/cephcsi:v3.3.1"
+  # ROOK_CSI_CEPH_IMAGE: "quay.io/cephcsi/cephcsi:v3.4.0"
   # ROOK_CSI_REGISTRAR_IMAGE: "k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0"
   # ROOK_CSI_RESIZER_IMAGE: "k8s.gcr.io/sig-storage/csi-resizer:v1.2.0"
   # ROOK_CSI_PROVISIONER_IMAGE: "k8s.gcr.io/sig-storage/csi-provisioner:v2.2.2"
diff --git a/pkg/operator/ceph/csi/spec.go b/pkg/operator/ceph/csi/spec.go
index 573aa9d38..9374380a7 100644
--- a/pkg/operator/ceph/csi/spec.go
+++ b/pkg/operator/ceph/csi/spec.go
@@ -108,7 +108,7 @@ var (
 // manually challenging.
 var (
 	// image names
-	DefaultCSIPluginImage         = "quay.io/cephcsi/cephcsi:v3.3.1"
+	DefaultCSIPluginImage         = "quay.io/cephcsi/cephcsi:v3.4.0"
 	DefaultRegistrarImage         = "k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.2.0"
 	DefaultProvisionerImage       = "k8s.gcr.io/sig-storage/csi-provisioner:v2.2.2"
 	DefaultAttacherImage          = "k8s.gcr.io/sig-storage/csi-attacher:v3.2.1"
diff --git a/pkg/operator/ceph/csi/version.go b/pkg/operator/ceph/csi/version.go
index 8f978b02c..685a488f2 100644
--- a/pkg/operator/ceph/csi/version.go
+++ b/pkg/operator/ceph/csi/version.go
@@ -25,15 +25,20 @@ import (
 )
 
 var (
-	//minimum supported version is 2.0.0
-	minimum = CephCSIVersion{2, 0, 0}
+	//minimum supported version is 3.0.0
+	minimum = CephCSIVersion{3, 0, 0}
 	//supportedCSIVersions are versions that rook supports
-	releaseV210          = CephCSIVersion{2, 1, 0}
-	releasev300          = CephCSIVersion{3, 0, 0}
 	releasev310          = CephCSIVersion{3, 1, 0}
 	releasev320          = CephCSIVersion{3, 2, 0}
 	releasev330          = CephCSIVersion{3, 3, 0}
-	supportedCSIVersions = []CephCSIVersion{minimum, releaseV210, releasev300, releasev310, releasev320, releasev330}
+	releasev340          = CephCSIVersion{3, 4, 0}
+	supportedCSIVersions = []CephCSIVersion{
+		minimum,
+		releasev310,
+		releasev320,
+		releasev330,
+		releasev340,
+	}
 	// omap generator is supported in v3.2.0+
 	omapSupportedVersions = releasev320
 	// for parsing the output of `cephcsi`


Test patch:
diff --git a/pkg/operator/ceph/csi/version_test.go b/pkg/operator/ceph/csi/version_test.go
index fb7d05b99..f09dacf1a 100644
--- a/pkg/operator/ceph/csi/version_test.go
+++ b/pkg/operator/ceph/csi/version_test.go
@@ -29,6 +29,7 @@ var (
 	testReleaseV320        = CephCSIVersion{3, 2, 0}
 	testReleaseV321        = CephCSIVersion{3, 2, 1}
 	testReleaseV330        = CephCSIVersion{3, 3, 0}
+	testReleaseV340        = CephCSIVersion{3, 4, 0}
 	testVersionUnsupported = CephCSIVersion{4, 0, 0}
 )
 
@@ -82,6 +83,11 @@ func TestIsAtLeast(t *testing.T) {
 	ret = testReleaseV330.isAtLeast(&testReleaseV300)
 	assert.Equal(t, true, ret)
 
+	// Test for 3.4.0
+	// Test version which is lesser
+	ret = testReleaseV340.isAtLeast(&testReleaseV330)
+	assert.Equal(t, true, ret)
+
 	// Test version which is greater (minor)
 	version = CephCSIVersion{3, 1, 1}
 	ret = testReleaseV300.isAtLeast(&version)
@@ -96,13 +102,13 @@ func TestIsAtLeast(t *testing.T) {
 func TestSupported(t *testing.T) {
 	AllowUnsupported = false
 	ret := testMinVersion.Supported()
-	assert.Equal(t, true, ret)
-
-	ret = testMinVersion.Supported()
-	assert.Equal(t, true, ret)
+	assert.Equal(t, false, ret)
 
 	ret = testVersionUnsupported.Supported()
 	assert.Equal(t, false, ret)
+
+	ret = testReleaseV340.Supported()
+	assert.Equal(t, true, ret)
 }
 
 func TestSupportOMAPController(t *testing.T) {