Change:
* **[stable]** Add support to use NFSv3, NFSv4, and GlusterFS on GCE/GKE GCI image based clusters. * **[beta]** Added support for mount options in persistent volumes. * **[alpha]** All in one volume proposal - a new volume driver capable of projecting secrets, configmaps, and downward API items into the same directory. <del> * **[alpha]** Flex volume API and Improved lifecycle (flexvolume). </del> ## Deprecations * Remove extensions/v1beta1 Jobs resource, and job/v1beta1 generator. ([#38614](https://github.com/kubernetes/kubernetes/pull/38614), [@soltysh](https://github.com/soltysh))

Discussion:
has the pull request for the proposal. The goal of this proposal is to extend and enhance the existing flex volume storage plugin model (). It bridges the gap with existing in-tree plugins and defines a stable api. Existing flex volume plugin enables administrators/vendors to create out-of-tree volume plugins, i.e., plugins which do not reside in Kubernetes project. But, it has the following gaps. It lacks support for the latest enhancements to in-tree plugins like dynamic provisioning and centralized attach-detach controller It does not define a stable driver api for administrators/vendors to work. The current flex volume plugin deployment model is not suitable for environments like GCE & CoreOS, where access to the root file system is restricted. And in other environments the overall user experience is poor, as it requires manual installation of the plugin on every node and restart of kubelets & controller manager. This issue is covered in detail in another PR. The goals of this proposal are Define new stable driver API. Add support for dynamic provisioning and centralized attach-detach controller and The biggest advantage of flex volume is it’s simplicity. It is just an adapter around the internal api. We will keep it simple in the new design and try to follow the same adapter pattern.This proposal does not the address the poor installation model of existing model. It is covered in another PR. This proposal uses the existing ‘exec’ plugin model. A rest callout model will be covered in another PR. Plugin: refers to flex volume plugin. Plugin Driver api: API calls defined by flex volume plugin. Plugin Driver: Out-of-tree driver/plugin which implements plugin driver api. In-tree plugin: Plugins which are included in Kubernetes project/source tree. Existing flex volume plugin driver api are in alpha phase and are subject to change between kubernetes versions. This section proposes stable api for plugin driver writers to work with. Expose the existing in-tree plugin API as driver API. For more details on existing in-tree plugin API please refer to In-tree plugins are already using these API. Simple and clean interface.No standard request & response API objects. No versioning support. Does not follow REST model. Many call outs. For full support, a plugin has to implement 17 APIs. This was discussed in the storage-sig face to face and in lieu the following is proposed. Proposes fewer APIs with stable request & response API objects.

Has standard request & response API objects. Supports versioning. Follows REST model. Fewer calls. Simple and clean interface.New API objects. This call probes the plugin for it’s capabilities and supported options. Supported options are used to validate and reject pod spec. This is executed from both the Controller-manager and Kubelet. On success: On failure: This call creates a volume. It is executed from Controller-manager. On success: On failure: This call deletes a volume. It is executed from Controller-manager. On success: On failure: This call attaches a volume to a remote host or local host. It can be executed from Controller-manager/Kubelet depending on whether Controller-attach-detach is enabled or not.. This is only valid for drivers which support attach & detach. The following additional options are passed to the plugin as part of options map in FlexVolumeSource. On success: On failure: This call detaches a volume to a remote host or local host. It can be executed from Controller-manager/Kubelet depending on whether Controller-attach-detach is enabled or not.. This call only valid for drivers which support attach & detach. The following additional options are passed to the plugin as part of options map in FlexVolumeSource. On success: On failure: This call mounts a volume on the node. It is executed from Kubelet. This is only valid for plugins which do not support attach/detach to a node. Example: NFS/CIFS. For volumes which support attach/detach, this will use the default logic to bind mount. The following additional options are passed to the plugin as part of options map in FlexVolumeSource. On success: On failure: This call unmounts a volume on the node. It is executed from Kubelet. This is only valid for plugins which do not support attach/detach to a node. Example: NFS/CIFS. On success: On failure: This call gets the metrics of a volume. It is executed from Kubelet. On success: On failure: The new driver probe function has support to query the capabilities and if the driver has of ‘dynamic provisioning’ capability, Flex volume plugin will support dynamic provisioning by implementing the ‘Provisioner’ and ‘Deleter’ interfaces.

The new driver probe function has support to query the capabilities and if the driver has of ‘attachment’ capability, Flex volume plugin will support dynamic provisioning by implementing the ‘Provisioner’ and ‘Deleter’ interfaces. PR by () would be leveraged for this change.
Please review.
On 09/13/2016 12:35 PM, Chakravarthy Nelluri wrote:
For volumes that are able to manage credentials and endpoints , the proposed APIs should work. For volumes that want to access or create secret, endpoint, or service, it is unclear how the proposed API can support them.
Thanks for the feedback. The secrets are part of FlexVolumeSource. If we have a good use case for endpoint or service information, we can standardize it or the user can pass it as an option.
One thing we are working on now is to refactor glusterfs's endpoint (and its service) handling
Am I misreading here ? The proposal says requests to flexvolume drivers will use REST, but then the callouts are "driver executable params" For REST based flexvolume drivers, shouldnt the call out be a GET/PUT/POST with a spec for request body and response body (in json or something) ? Also, there are some typos ... e.g. Detach description talks about attach.
Just like "Delete a volume", are we missing a "Recycle a volume" API ? Also, how does a flexvolume driver express (in Probe) that it supports deleter and/or recycler ? Is this in the SupportedOptions field ? If not, are we missing two bools in FlexVolumeDriverCapabilities type ?
Why is the actual driver binary required in the FlexVolumeDriverSpec ? If API calls are REST, shouldn't that be a REST endpoint ?
this defines the REST semantic API types and the callouts are still based on exec model. HTTP based interface will be in another subsequent proposal.
Thanks I fixed the typos.
+1 to complete Attachment with another flag like I'm for limiting the call to one, well-defined object, and avoid as much a possible to use the options map, which I think shoudl be reserved for driver's "blackbox" data.
Driver caps structure as a consequence:
I'm not sure we need ObjectMeta and TypeMeta in the results from the calls.
I'm for using FlexVolumeMount as a parameter for "metrics".

The LocalAttachment field has some consequences, I wonder what your opinion is regarding them. means each driver can choose independently if they want LocalAttachment (or not) an impact on the enable-controller-attach-detach command line parameter to kubelet. (That flag as long as I understand is for that node, and is not driver specific). if we decide to dump that flag, and let the LocalAttachment flag decide the attachment mode, what happens to non-flexvolume drivers.
looks weird to have it at a wider level than the driver; we may which a Ceph RBD volume and GCE volume on the same host for instance. should specify how they work, this allow to factorize more complex code like RPC mecanisms in Kubernetes itself, providing more service to the drivers, thus helping implementors.
We agree on 1. I dont think we can remove the flag for 2 so easily, we still need to support existing non-flexvolume drivers. I am confused about what you want to say in (3). Please explain (3).
On 2, we can of course keep the flag for now. Then, on 3, other drivers may or may not move, but I'd be for a generic support in Kubernetes, so the complexity is done once for all.
For (2), if we are keeping the flag, what is the expectation when that flag is specified, but the driver specifies LocalAttachment is true ? Obviously, one of them should win :) I think it should be the driver (since that flag will be deprecated soon, it can be overridden by flexvolume drivers written using this new proposal) ? We should get somebody who works in kubelet code to okay this .....
(2) I think LocalAttachment should win; the flag is for core drivers :) We may give it as an hint in the probe call, but that looks unclean and overkill to me. So we agree.
The driver flag should win in the long term. and are the right persons to comment if we need to use Kubelet flag for the short term. If not, we will use the driver flag for the short term too.
there are indeed some cases we should take a deep look. If the driver doesn't support remote attach but kubelet and controller master are enabled to attach volume remotely, we should see if the driver can reject the call from controller and let kubelet attach the volume.

It might not be possible for a driver to be used in whatever mode is available. It depends on how the storage device behaves/works. For example, a driver written for controller attach will probably not have storage credentials being sent to the node. So just asking the driver to do a kubelet attach instead of controller attach will not work, unless the driver itself is written to be multimodal. I think the kubelet flag for attach/detach was fine for a start, but it is too coarse now. In a k8s cluster where multiple volume drivers are present, this decision should be made on a per driver basis. The Probe API is a step in that direction, we should eventually phase out the global flag (and the kubelet attach/detach commandline parameter). Do you agree ? If not, your opinions please ?
It's unclear: are these types in the main kubernetes API or some new API that is just flex drivers? Calling it REST is confusing - this is just structured input and output. It might be simpler to digest if you just called it that? REST implies resources which act like nouns and have different verbs. This is closer to JSON RPC.
Thanks These API are just for Flex drivers. What would be the best place for the external interfaces like this? The API types are REST objects/resources. Attach & Detach verbs are used in callouts in exec model, similar to how we do "kubectl scale --replicas..." for replication controller. The plan is to use the same API objects for HTTP interface. The attach & detach will change to POST/PUT on corresponding URL.
Talked to Tim. We want to give it a shot to bring it closer to Docker API. I will update the proposal.
the credential issue is being addressed in
w.r.t. secrets - can I confirm that the following use case is one that this proposal would support?: We want to supply unique TLS certificates to every pod in a Deployment. We are investigating using volumeClaimTemplates to dynamically provision secret volumes, one per pod. The provisioner would need the pod dns name to set the CN in the certificate, and would create a certificate, then load that into a secret object - we'd want to return a reference to the secret object, so the volume mount on the minion would follow the normal secret behaviours.
we cannot support this use case.

Provisioner creates volumes when a claim is created & bound. There is no pod at that point of time. We cannot pass the pod DNS name to the provisioner.
ah - thats a shame. Is that the only limit? e.g. another way we could do it is to place a unique token in the secret, and use that to exchange for a certificate - only that pod would know the token, and the certificate service could introspect the pod to see the claim metadata.
AFAIK we do not have a way to pass the pod specific or per-volume specific parameters to dynamic provisioning infra (it has nothing to do with flex volume). The only way is to manually create persistent volumes with pod specific parameters, every time you want one. Between, you will not be able to return secret objects, provisioner only handles persistent volumes. You can still have your own custom logic to populate the secrets during the plugin mount call using flex volume.
I agree with the right place seems to be the mount call for a flexvolume. But I think it would be much simpler to have a container creating it in a emptyDir volume.
perhaps I should open a new issue, to avoid taking this one sideways? I'm obviously missing something - how do I make sure I've only handed out the cert to the desired pod if the pod is just using an emptyDir volume itself? Presumably the pod is then asking the CA for the cert directly - but it has nothing with which it can proove its identity except its IP address - and if we're assuming the network is unspoofable we are perhaps not needing TLS in the first place :). So, ok - we make a trivially empty secret object [ or perhaps your note about only handling pv's means we don't create a secret object at all, but just some placeholder pv - I guess we need a new pv type ?] upon request in the provisioning call, and then in the mount call we call out to our CA, as at that point we know the pod identity, and our agents callout would use credentials stored in the node itself - presumably we can drop them on disk as a salt pillar or some such.
for now, the discussion volume on this is ok I think, and if we get a conclusion it may help API design; but if it begins to be long debate then yes, I'm in favor of another issue.

For the initialization pod auth, I was thinking of the service account (): that should allow you to rely on the existing kubernetes PKI and access a Secret object (or whatever else).
Update: Docker API is very minimalistic as of today. We were interested in extending the API to have an uniform interface between Docker & Kubernetes. Talking to Docker, learnt that the existing volume plugins approach of centralized controller (and remote attach/detach calls) might not be a fit for Docker & Swarm architecture. Given that and the slew of new features we have in pipeline(ex: snapshots, local storage) and the new API extensions we might have to add to support them, we decided to implement our own API model. Working on the review comments on the existing proposal. Will update the proposal ASAP.
IIUC that sounds doable :). If you need credentials to talk to your CA, you can use Kubernetes secrets for it. Flex volume will pass the secret in the call outs.
Updated the proposal in PR
are service accounts templated in Deployments? Seems like every pod gets the same service account - default ? We'd be trusting no-one else manages to spawn pods in the same namespace AIUI - and I'm not saying thats a problem, just trying to make sure I have a clear understanding of the trust boundaries we're going to be establishing.
Yes, the trust boundaries are at the namespace level here. If you want it at the pod level, the flexvolume approach would be ok (use the mount call that know the target pod). IIUC, in your case, it seems that the pods must not access the API anyway because you have a very low level of trust in them. If they can't access the API, they won't be able to read the secrets in their namespace. In this case, I think you're more willing to create a "secret provisionner" than a full flexvolume plugin; it would be simpler to design and maintain, and will be a single pod in the cluster whereas a flexvolume plugin would require a daemon set (1 pod per host).
PTAL and let me know your feedback. I have changed the API to gRPC API.
What's the status of this proposal?
this is getting superseded by CSI.