Change:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: fsgroup is only intended for RWO volumes because it recursively changes the permissions of all the directories in the volume. This can cause problems for nfs-type volumes because multiple pods could access the same volume and have the permissions changed out from underneath. What you expected to happen: Determine whether or not the plugin supports fsgroup either via: access mode. This is still a bit ambiguous for nfs volume types, that can theoretically support rwo, especially since Kubernetes does not do any sort of access mode enforcement. a new csi capability
CC
Are you saying that CSI internal driver should check driver capability for RWO, if supported, only then apply value ? What would the be ?
Yes one possible way is to infer fsgroup is supported for all rwo volumes, which may not be true? Like a nfs volume technically supports rwo but not fsgroup. So the other option is to make it a new CSI capability. I'm not sure exactly sure how best to describe it though, maybe something about volume ownership management.
Will it work if is not set then don't chown the ownership? I would think that for shared file system types(nfs, glusterfs) - won't be set. For raw block storage too - I think we shouldn't have to do this.
what do you mean by ? Shouldnt have to solve this issue ?
I meant - which is obvious. I just meant that the entire mechanism of changing ownership of files should apply only to volume types with block storage filesystems. I am just wondering if we can get away with detecting fsType and not have to introduce new capability in CSI.
agreed, do not map well to current csi.Capabilities enums which makes it hard to determine RWO when mounting. Perhaps, instead of a new capability, we revisit the existing capabilities to see if there are any way RWO can be reliably inferred from returned capabilities. Currently, this is what the code is doing which we knew needed to be revisited:
Users may not fully understand what fsgroup is and set it anyway, or a default psp with some fsgroup may be defined in which it will automatically be set on all pods. The first case we can attribute to user error, but the second case may not have a good workaround and could cause permission issues on rwx volumes. So I think we do need to solve this.
Yes, agreed this has to be solved. As you pointed out user error or improperly configured PSP can set value that does not match the correct attribute, causing permission issues. Here is what I think can be done without new CSI capabilities: , then do not apply fsgroup then: , apply fsgroup , do not apply fsgroup to void stated issue then: , apply fsgroup I think that may help deduce when to apply fsgroup.
If mount reference count <= 1, apply fsgroup If mount reference count 1, do not apply fsgroup to void stated issue A RWM volume (say NFS) can be mounted on multiple nodes. Do you have reliable count of mounts across all nodes?
In addition, that would be a change in semantics to how we support rwx volumes today. If we don't have a new CSI capability, then I think the best course of action is to only apply fsgroup for RWO volumes. Some questions remain: Do all RWO volumes support fsgroup? RWX is a superset of RWO. So if a user requests a RWO volume, it could still be satisfied by a RWX volume type. Can we detect this and still not apply the fsgroup even though the volume capability will say RWO?
If there's not a reliable way to enumerate node mounts (thinking about it, for CSI external drivers, the answer is NO). Theres is no reliable way to have k8s autocorrect a bad fsgroup/PV.AccessMode combo. So we would have to rely on the config provided by user/admin. I have come to similar conclusion as -- mainly, apply fsgroup only to volumes with access modes. The other k8s specified access modes, , cannot be reliably applied. Even if we were to introduce additional plugin capabilities that the CO can query (from driver) to find out what modes can be applied, there would still be the possibility of user/admin mis-configuration. Probably not, Is this where you think an additional capability would help ( ) That is a good point. Can't think of a way to detect/guess RWX when RWO is specified, this without the driver providing that info. Can we adopt the rule until we decide what capabilities would look like ?
I will just try to summarize the conversation from Slack. We can safely assume that only applies to block storage type with file system on top of it. So for all - volumes that need to have called MUST have a file system() on them. Is that assumption correct? If it is - can we not use presence of field or we can query the local volume for fs type (like how we query selinux relabelling capability) and call only when - volume is RW and has a valid block storage file system type. Will this not solve our use case? EDIT: renamed to .
We'll need to make it clear that is only for block-based storage systems, and the field cannot be reused for some multi-protocol file server (ie, nfsv3, nfsv4, smb, etc). For that case, then the plugin will need to expose its own new storageclass parameter + volume attribute to specify protocol.
Can we not use to get fsType - and use that information for calling ? It may be problematic to rely entirely on present inside csi source.
the only problem with is that in CSI volume info is intentionally opaque to the CSI code inside k8s. All mount logic is delegated to the driver.
Instead of a new capability or trying to deduce how best to apply fsgroup, why not pass the to the driver and let it decide how to apply it (either as a attribute for the mount or top level request param)? Right now, all mount logic has been delegated to the external CSI driver except for applying permission. Thoughts ?
that is true but we can still inspect file system on the mounted path because by the time gets called volume is already mounted. This should be no different than how selinux support is determined. what you think?
is this approved for 1.12 milestone? Thank you for working on this,
still discussing how best to approach this. It would be a 1.12.
[MILESTONENOTIFIER] Milestone Issue: Up-to-date for process <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
Approved for 1.12
Based on previous post, this is a summary: If then is not applied (because it could be an indication of a nonblock fs, or error) Done is provided if fstype is invalid Done if fstype is valid and then apply fsgroup logic else ignore fsGroup Done
- any updates on ?
Waiting for a review and approval.
you have a outstanding comment on the PR. Can you please answer that?
How does this approach relate to inline CSI volumes? There's no PersistentVolume involved, and hence no way to specify an access mode. , but my driver ignores it (always using tmpfs). I considered having my CSI driver read the mounting pod's spec to see if it has a security context with "fsGroup" set, since that detail is not supplied to the driver, as . My goal is to have the volume mounted with the files on it readable by the container's user, but there's no way for my driver to know who that user will be. Am I supposed to just use 0444 for the ownership bits, to allow everyone to read the files?

Discussion:
} // apply volume ownership <del> if !c.readOnly && fsGroup != nil { err := volume.SetVolumeOwnership(c, fsGroup) if err != nil { // attempt to rollback mount. glog.Error(log("mounter.SetupAt failed to set fsgroup volume ownership for [%s]: %v", c.volumeID, err)) glog.V(4).Info(log("mounter.SetupAt attempting to unpublish volume %s due to previous error", c.volumeID)) if unpubErr := csi.NodeUnpublishVolume(ctx, c.volumeID, dir); unpubErr != nil { glog.Error(log( "mounter.SetupAt failed to unpublish volume [%s]: %v (caused by previous NodePublish error: %v)", c.volumeID, unpubErr, err, )) return fmt.Errorf("%v (caused by %v)", unpubErr, err) } </del> <ins> // The following logic is derived from https://github.com/kubernetes/kubernetes/issues/66323 // if fstype is "", then skip fsgroup (could be indication of non-block filesystem) // if fstype is provided and pv.AccessMode == ReadWriteOnly, then apply fsgroup </ins> <del> if unmountErr := removeMountDir(c.plugin, dir); unmountErr != nil { glog.Error(log( "mounter.SetupAt failed to clean mount dir [%s]: %v (caused by previous NodePublish error: %v)", dir, unmountErr, err, )) return fmt.Errorf("%v (caused by %v)", unmountErr, err) } </del> <ins> err = c.applyFSGroup(fsType, fsGroup) if err != nil { // attempt to rollback mount. fsGrpErr := fmt.Errorf("applyFSGroup failed for vol %s: %v", c.volumeID, err) if unpubErr := csi.NodeUnpublishVolume(ctx, c.volumeID, dir); unpubErr != nil { glog.Error(log("NodeUnpublishVolume failed for [%s]: %v", c.volumeID, unpubErr)) return fsGrpErr } </ins> <del> return err </del> <ins> if unmountErr := removeMountDir(c.plugin, dir); unmountErr != nil { glog.Error(log("removeMountDir failed for [%s]: %v", dir, unmountErr)) return fsGrpErr </ins> } <del> glog.V(4).Info(log("mounter.SetupAt sets fsGroup to [%d] for %s", *fsGroup, c.volumeID)) </del> <ins> return fsGrpErr </ins> } glog.V(4).Infof(log("mounter.SetUp successfully requested NodePublish [%s]", dir))

return nil } <ins> // applyFSGroup applies the volume ownership it derives its logic // from https://github.com/kubernetes/kubernetes/issues/66323 // 1) if fstype is "", then skip fsgroup (could be indication of non-block filesystem) // 2) if fstype is provided and pv.AccessMode == ReadWriteOnly and !c.spec.ReadOnly then apply fsgroup func (c *csiMountMgr) applyFSGroup(fsType string, fsGroup *int64) error { if fsGroup != nil { if fsType == "" { glog.V(4).Info(log("mounter.SetupAt WARNING: skipping fsGroup, fsType not provided")) return nil } accessModes := c.spec.PersistentVolume.Spec.AccessModes if c.spec.PersistentVolume.Spec.AccessModes == nil { glog.V(4).Info(log("mounter.SetupAt WARNING: skipping fsGroup, access modes not provided")) return nil } if !hasReadWriteOnce(accessModes) { glog.V(4).Info(log("mounter.SetupAt WARNING: skipping fsGroup, only support ReadWriteOnce access mode")) return nil } if c.readOnly { glog.V(4).Info(log("mounter.SetupAt WARNING: skipping fsGroup, volume is readOnly")) return nil } err := volume.SetVolumeOwnership(c, fsGroup) if err != nil { return err } glog.V(4).Info(log("mounter.SetupAt fsGroup [%d] applied successfully to %s", *fsGroup, c.volumeID)) } return nil } </ins> // isDirMounted returns the !notMounted result from IsLikelyNotMountPoint check func isDirMounted(plug *csiPlugin, dir string) (bool, error) { mounter := plug.host.GetMounter(plug.GetPluginName())

MounterSetUpTests(t, false) }) } <ins> func TestMounterSetUpWithFSGroup(t *testing.T) { fakeClient := fakeclient.NewSimpleClientset() plug, tmpDir := newTestPlugin(t, fakeClient, nil) defer os.RemoveAll(tmpDir) testCases := []struct { name        string accessModes []api.PersistentVolumeAccessMode readOnly    bool fsType      string setFsGroup  bool fsGroup     int64 }{ { name: "default fstype, with no fsgroup (should not apply fsgroup)", accessModes: []api.PersistentVolumeAccessMode{ api.ReadWriteOnce, }, readOnly: false, fsType:   "", }, { name: "default fstype  with fsgroup (should not apply fsgroup)", accessModes: []api.PersistentVolumeAccessMode{ api.ReadWriteOnce, }, readOnly:   false, fsType:     "", setFsGroup: true, fsGroup:    3000, }, { name: "fstype, fsgroup, RWM, ROM provided (should not apply fsgroup)", accessModes: []api.PersistentVolumeAccessMode{ api.ReadWriteMany, api.ReadOnlyMany, }, fsType:     "ext4", setFsGroup: true, fsGroup:    3000, }, { name: "fstype, fsgroup, RWO, but readOnly (should not apply fsgroup)", accessModes: []api.PersistentVolumeAccessMode{ api.ReadWriteOnce, }, readOnly:   true, fsType:     "ext4", setFsGroup: true, fsGroup:    3000, }, { name: "fstype, fsgroup, RWO provided (should apply fsgroup)", accessModes: []api.PersistentVolumeAccessMode{ api.ReadWriteOnce, }, fsType:     "ext4", setFsGroup: true, fsGroup:    3000, }, } for i, tc := range testCases { t.Logf("Running test %s", tc.name) volName := fmt.Sprintf("test-vol-%d", i) pv := makeTestPV("test-pv", 10, testDriver, volName) pv.Spec.AccessModes = tc.accessModes pvName := pv.GetName() spec := volume.NewSpecFromPersistentVolume(pv, tc.readOnly) if tc.fsType != "" { spec.PersistentVolume.Spec.CSI.FSType = tc.fsType } mounter, err := plug.NewMounter( spec, &api.Pod{ObjectMeta: meta.ObjectMeta{UID: testPodUID, Namespace: testns}}, volume.VolumeOptions{}, ) if err != nil { t.Fatalf("Failed to make a new Mounter: %v", err) } if mounter == nil { t.Fatal("failed to create CSI mounter") } csiMounter := mounter.(*csiMountMgr) csiMounter.csiClient = setupClient(t, true) attachID := getAttachmentName(csiMounter.volumeID, csiMounter.driverName, string(plug.host.GetNodeName())) attachment := makeTestAttachment(attachID, "test-node", pvName) _, err = csiMounter.k8s.StorageV1beta1().VolumeAttachments().Create(attachment) if err != nil { t.Errorf("failed to setup VolumeAttachment: %v", err) continue } // Mounter.SetUp() var fsGroupPtr *int64 if tc.setFsGroup { fsGroup := tc.fsGroup fsGroupPtr = &fsGroup } if err := csiMounter.SetUp(fsGroupPtr); err != nil { t.Fatalf("mounter.Setup failed: %v", err) } //Test the default value of file system type is not overridden if len(csiMounter.spec.PersistentVolume.Spec.CSI.FSType) != len(tc.fsType) { t.Errorf("file system type was overridden by type %s", csiMounter.spec.PersistentVolume.Spec.CSI.FSType) } // ensure call went all the way pubs := csiMounter.csiClient.(*fakeCsiDriverClient).nodeClient.GetNodePublishedVolumes() if pubs[csiMounter.volumeID].Path != csiMounter.GetPath() { t.Error("csi server may not have received NodePublishVolume call") } } } </ins> func TestUnmounterTeardown(t *testing.T) { plug, tmpDir := newTestPlugin(t, nil, nil)

sanitizedSpecVolID := kstrings.EscapeQualifiedNameForDisk(specVolID) return path.Join(host.GetVolumeDevicePluginDir(csiPluginName), sanitizedSpecVolID, "data") } <ins> // hasReadWriteOnce returns true if modes contains v1.ReadWriteOnce func hasReadWriteOnce(modes []api.PersistentVolumeAccessMode) bool { if modes == nil { return false } for _, mode := range modes { if mode == api.ReadWriteOnce { return true } } return false } </ins>