Code change:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: CSI Plugin calls to get node ID information (see ). If that call is not implemented by driver, registration with kubelet will fail. What you expected to happen: Per CSI Spec, the deprecated call should be supported until CSI 1.0 is released (see ). Therefore if a by Kubernetes fails (because it is not implemented by the driver), we should fall back and try call instead. How to reproduce it (as minimally and precisely as possible): Use any CSI driver that implements but not -- says OpenSDS driver is currently hitting this, and may be used for repro. Anything else we need to know?: When CSI moves to 1.0 we can drop this fall back logic. If a fix is available within , we can include it, but 1.12 release should NOT block on this. /sig storage Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
please verify this is the issue you hit. Specifically see if you can find in your kubelet logs.
CC who volunteered to take a look
I don't see "error during CSI NodeGetInfo" in my kubelet logs. I don't see glog.Infof(log("Register new plugin with name: %s at endpoint: %s", pluginName, endpoint)) so it seems that RegisterPlugin is not called. I do see a message: "Plugin Watcher Start at /var/lib/kubelet/plugins". I don't know why RegisterPlugin is not called. I'll continue to check.
Here's an update. I implemented NodeGetInfo. I followed documentation about the Kubelet Plugin Watcher: https://kubernetes- to get my plugin registered. After that I don’t see the “driver name csi-opensdsplugin not found in the list of registred CSI drivers” any more. I see that now RegisterPlugin gets called and I see the following message in I0914 14:44:50. ] Register new plugin with name: csi-opensdsplugin at endpoint: /var/lib/kubelet/plugins/csi- So this confirmed that missing NodeGetInfo is the problem. I still have problems but I think that’s a config issue on my side. I the following as cluster role rule for attacher and node: apiGroups: [""] resources: ["csidrivers"] verbs: ["get", "list", "watch", "update"] However I’m still getting the following errors in the logs: I0914 16:12:31. ] Listing and watching v1alpha1.CSIDriver from E0914 16:12:31. ] Failed to list v1alpha1.CSIDriver: is forbidden: User "system:serviceaccount:kube-system:attachdetach-controller" cannot list resource "csidrivers" in API group "" at the cluster scope kube-controller- I0914 16:11:51. ] Listing and watching v1alpha1.CSIDriver from E0914 16:11:51. ] Failed to list v1alpha1.CSIDriver: is forbidden: User "system:node:127.0.0.1" cannot list resource "csidrivers" in API group "" at the cluster scope
Did more investigation on this. Here's an update. I set the following feature gate to true on my local cluster: alpha: CSICRDAutoInstall CSISkipAttach CSIPodInfo I got some new errors in and kube-controller- I0916 07:22:06. ] Listing and watching v1alpha1.CSIDriver from E0916 07:22:06. ] Failed to list v1alpha1.CSIDriver: the server could not find the requested resource (get ) I0916 07:25:24. ] Listing and watching v1alpha1.CSIDriver from E0916 07:25:24. ] Failed to list v1alpha1.CSIDriver: the server could not find the requested resource (get ) In addition, I found the following in kube-controller- E0916 07:22:01. ] failed to create CSIDrivers CRD: &v1beta1.CustomResourceDefinition{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"", GenerateName:"", Namespace:"", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(time.Location)(nil)}}, DeletionTimestamp:(v1.Time)(nil), DeletionGracePeriodSeconds:(int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Initializers:(v1.Initializers)(nil), Finalizers:[]string(nil), ClusterName:""}, Spec:v1beta1.CustomResourceDefinitionSpec{Group:"", Version:"", Names:v1beta1.CustomResourceDefinitionNames{Plural:"", Singular:"", ShortNames:[]string(nil), Kind:"", ListKind:"", Categories:[]string(nil)}, Scope:"", Validation:(v1beta1.CustomResourceValidation)(nil), Subresources:(v1beta1.CustomResourceSubresources)(nil), Versions:[]v1beta1.CustomResourceDefinitionVersion(nil), AdditionalPrinterColumns:[]v1beta1.CustomResourceColumnDefinition(nil)}, Status:v1beta1.CustomResourceDefinitionStatus{Conditions:[]v1beta1.CustomResourceDefinitionCondition(nil), AcceptedNames:v1beta1.CustomResourceDefinitionNames{Plural:"", Singular:"", ShortNames:[]string(nil), Kind:"", ListKind:"", Categories:[]string(nil)}, StoredVersions:[]string(nil)}}, err: &errors.StatusError{ErrStatus:v1.Status{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ListMeta:v1.ListMeta{SelfLink:"", ResourceVersion:"", Continue:""}, Status:"Failure", Message:" is forbidden: User "system:serviceaccount:kube-system:attachdetach-controller" cannot create resource "customresourcedefinitions" in API group "" at the cluster scope", Reason:"Forbidden", Details:(*v1.StatusDetails)(0xc4212af560), Code:403}} So I made this change to add RBAC rule for customresourcedefinitions: After that, my CSI plugin is functional. I can create/attach/mount volumes successfully now.
/cc
(As we discussed offline) the latest code in master no longer has CSICRDAutoInstall, CSISkipAttach, CSIPodInfo flags. New CSI features are now flag gated under CSINodeInfo and CSIDriverRegistry flags (both alpha and disabled by default). KubeletPluginsWatcher must be enabled as well, but that's in beta now so it's enabled by default. Since this PR merged: , CRDs must be installed manually, and you can find an example of it here: (note they have to be created with --validation=false kubectl flag for the time being). This PR has not been cherry-picked to 1.12.0-rc.0 yet.
Thanks Tested latest code. If NodeGetInfo is implemented, we just need to make necessary changes when deploying driver-registrar sidecar container so KubeletPluginsWatcher can work. NodeGetInfo is needed for kubelet to register CSI plugins. Also tested with and without "CSINodeInfo=true,CSIDriverRegistry=true" and manually installed CRDs using when they are enabled.
I have the same problem when deploying my CSI plugin based on CSI v0.2.0 on Kubernetes v.1.12.0. Daemonset pods of CSI plugin cannot launch successfully. Kubelet logs:
Considering Kubernetes v1.12 is already out, and we planned to deprecate the old call in CSI 1.0/Kubernetes 1.13 anyway, we'll pull of the bandaid and deprecate it now. If you are running Kubernetes 1.13, make sure your CSI driver implements the .
looks like you might be running in to a different issue: Kubernetes v1.13 also enables Kubelet device plugin registration by default. Before upgrading to v1.13, ensure the CSI sidecar container for your CSI driver is configured to handle plugin registration (set the parameter on to expose a new unix domain socket to handle Kubelet Plugin Registration). From
Hi Because goes Beta and is enabled by default in Kubernetes v1.12. After disabling this feater gate, I can run CSI plugin based on CSI spec v0.2.0 in Kubernetes v1.12.2. When Kubelet enable , Kubelet would call gRPC call to register CSI plugin. gRPC call was first proposed at CSI spec v0.3.0. Therefore, user should disable feature gate when running CSI plugin based on CSI spec v0.2.0 in Kubernetes v1.12.
I have implement the NodeGetInfo call. But it still has this issue. My kubernetes version is 1.14.3.

Related discussion:
text: * kube-apiserver: the `Priority` admission plugin is now enabled by default when using `--enable-admission-plugins`. If using `--admission-control` to fully specify the set of admission plugins, the `Priority` admission plugin should be added if using the `PodPriority` feature, which is enabled by default in 1.11. ([#65739](https://github.com/kubernetes/kubernetes/pull/65739), [@liggitt](https://github.com/liggitt)) * The `system-node-critical` and `system-cluster-critical` priority classes are now limited to the `kube-system` namespace by the `PodPriority` admission plugin. ([#65593](https://github.com/kubernetes/kubernetes/pull/65593), [@bsalamat](https://github.com/bsalamat)) * kubernetes-worker juju charm: Added support for setting the --enable-ssl-chain-completion option on the ingress proxy.  "action required": if your installation relies on supplying incomplete certificate chains and using OCSP to fill them in, you must set "ingress-ssl-chain-completion" to "true" in your juju configuration. ([#63845](https://github.com/kubernetes/kubernetes/pull/63845), [@paulgear](https://github.com/paulgear)) <ins> * In anticipation of CSI 1.0 in the next release, Kubernetes 1.12 calls the CSI `NodeGetInfo` RPC instead of `NodeGetId` RPC. Ensure your CSI Driver implements `NodeGetInfo(...)` before upgrading to 1.12. [@saad-ali](https://github.com/kubernetes/kubernetes/issues/68688) * Kubernetes 1.12 also enables [Kubelet device plugin registration](https://github.com/kubernetes/features/issues/595) feature by default. Before upgrading to 1.12, ensure the `driver-registrar` CSI sidecar container for your CSI driver is configured to handle plugin registration (set the `--kubelet-registration-path` parameter on `driver-registrar` to expose a new unix domain socket to handle Kubelet Plugin Registration). </ins> ### Other notable changes