Change:
if kubeClient == nil { return nil, fmt.Errorf("glusterfs: failed to get kube client to initialize mounter") } ep, err := kubeClient.Core().Endpoints(ns).Get(epName, metav1.GetOptions{}) <del> if err != nil && errors.IsNotFound(err) { claim := spec.PersistentVolume.Spec.ClaimRef.Name checkEpName := dynamicEpSvcPrefix + claim if epName != checkEpName { return nil, fmt.Errorf("failed to get endpoint %s, error %v", epName, err) } glog.Errorf("glusterfs: failed to get endpoint %s[%v]", epName, err) if spec != nil && spec.PersistentVolume.Annotations[volumehelper.VolumeDynamicallyCreatedByKey] == heketiAnn { class, err := volutil.GetClassForVolume(plugin.host.GetKubeClient(), spec.PersistentVolume) if err != nil { return nil, fmt.Errorf("glusterfs: failed to get storageclass, error: %v", err) } cfg, err := parseClassParameters(class.Parameters, plugin.host.GetKubeClient()) if err != nil { return nil, fmt.Errorf("glusterfs: failed to parse parameters, error: %v", err) } scConfig := *cfg cli := gcli.NewClient(scConfig.url, scConfig.user, scConfig.secretValue) if cli == nil { return nil, fmt.Errorf("glusterfs: failed to create heketi client, error: %v", err) } volumeID := dstrings.TrimPrefix(source.Path, volPrefix) volInfo, err := cli.VolumeInfo(volumeID) if err != nil { return nil, fmt.Errorf("glusterfs: failed to get volume info, error: %v", err) } endpointIPs, err := getClusterNodes(cli, volInfo.Cluster) if err != nil { return nil, fmt.Errorf("glusterfs: failed to get cluster nodes, error: %v", err) } // Give an attempt to recreate endpoint/service. _, _, err = plugin.createEndpointService(ns, epName, endpointIPs, claim) if err != nil && !errors.IsAlreadyExists(err) { glog.Errorf("glusterfs: failed to recreate endpoint/service, error: %v", err) return nil, fmt.Errorf("failed to recreate endpoint/service, error: %v", err) } glog.V(3).Infof("glusterfs: endpoint/service [%v] successfully recreated ", epName) } else { return nil, err } </del> <ins> if err != nil { glog.Errorf("glusterfs: failed to get endpoints %s[%v]", epName, err) return nil, err </ins> } <ins> glog.V(1).Infof("glusterfs: endpoints %v", ep) </ins> return plugin.newMounterInternal(spec, ep, pod, plugin.host.GetMounter(), exec.New()) }

Discussion:
several operations to the glusterfs mount operation that nodes are not permitted to perform: looks up arbitrary secrets based on storage class parameters. this will not succeed, as the node authorizer will only allow nodes access to secrets referenced by pods or persistent volumes used by the node attempts to create a service and endpoint using the node's API client. nodes do not have permission to create arbitrary endpoints and services. For both of those reasons, that recreate operation will fail when attempted from the node. The recreate operation should probably move to one of the storage-related controllers (provisioner or attach/detach) cc
Marking this with 1.7 milestone until we have a better idea of whether this is blocking or not for the 1.7 release.
CC Is the endpoint originally created by the user or by the volume plugin creating/deleting Endpoint objects? In addition to the concerns raised, having a volume create new API objects seems like a bad idea (abstraction layer violation?).
From on Slack: Unless there is justification otherwise, I will roll back the PR on Monday.
cc/
revert button doesn't work, have to do it manually
Revert PR is approved and awaiting merge (). Once that merges this issue can be closed.
Opened an E2E pr to catch this kind of behavior in the future. WIP, still exploring a method to reproduce and catch the behavior.