Change:
Is this a BUG REPORT or FEATURE REQUEST?: /kind bug This is more of an enhancement. What happened: The Plugin Watcher is setup to trigger on file activities when they occur in the subdirectory that it scans. During startup, the Watcher is passed (from within the Kubelet) the Kubelet's root plugin directory as a location to scan for newly plugins - Because the kubelet's plugins subdirectory is also used by existing in-tree drivers (including csi) as a location to store plugins artifacts, the Plugin Watcher is getting triggered by directory/file activities occurring in that subdirectory that has nothing to do with plugin registration as shown in the output below: What you expected to happen: The Plugin Watcher should not be getting triggered by file/directory activities not related to plugin registration/unregistration. One possibility is to have the Watcher scan one directory below plugins like or something similar How to reproduce it (as minimally and precisely as possible): Run plugin watcher with a CSI driver. cc
/sig node
I think you should only place the plugin socket file in the plugin registration directory but not the plugin data.
agree with we can only place the registration file in the specified path instead of placing all data into it. On the other hand, you can place data into the /var/lib/kubelet/plugins/registration even if registration created, that's a no-end loop.
As I mentioned in issue, existing in-tree volume plugins were already using that directory as a location to store storage operation artifacts (not just csi). Here is a shortened list of driver locations that use /var/lib/kubelet/plugins in their code already: And several more locations. The fix would be to have plugin watcher scan a subdirectory of plugins (say ) instead of plugins directly as it is doing today. The watcher functions as intended. Except now it picks up all directory activities whether related or not to plugin registration.
Well given the fact that the pluginwatcher is beta this is a bit of an issue. My understanding of the "beta" guarantees is that we want to be backwards compatible for all plugins published with the beta API. Meaning that even if we move the registration directory to a new location, we'll need to keep scanning the old directory (to pick up v1beta plugins). From a 10 000 feet point of view it seems easier to me if we move the in-tree volume plugin's data. Though it might be a lot of work. Another less painful option, but a bit grey is the following: Enforce plugins to end with this suffix is recommended for device plugins (though not enforced) and I believe is a convention on linux. This is a bit grey because we would still be breaking v1beta users but since we're moving a convention from "recommended" to "enforced" it seems "less" breaking to me (and I haven't seen device plugins not using that extension). What do you think ?
I like the suffix idea. It can be a recommendation (which csi is already doing).
Hmm, I've just deployed a new k8s cluster on 1.12.0 and noticed this repeating in the logs on a few of our pods. Just for my own sanity, is this something I should be worried about or can I ignore it until a fix is released?
you can ignore these errors. if it's a recommendation then we'll keep spamming the logs with all the data files. Another thing to keep in mind, it seems like CSI plugins puts a lot of data (and creates a lot of directories). It might be worth considering limiting the recursion to one level. I'm concerned we might exhaust the amount of fds by tracking the plugins data.
yeah CSI keeps track of volume operation metadata in the plugins dir. This probably will not change. I understand this is beta. However, besides CSI and Device plugin, are there any other implementers out there for this feature ? Would it be possible to propose a new location (i.e. <pluginsDir/reg/?
let me know if you open a PR for this as discussed.
thanks for open the issue with detail logs, i can see two problems by looking at logs and code. 1) plugin watcher trying to dial non-socket files. 2) too many logs generated (CREATE and DELETE). I a suggestion to fix of this issue we can fix those problems by: 1) check file mode when handle CREATE event, only attempt to dial file when file mode is socket. 2) increase log verbose level for both 'handleDeleteEvent()' and 'handleCreateEvent()' plugin watcher still has activity when any file create/delete under the directory, but they become light weighted (CREATE file will only trigger file mode check, DELETE file event will trigger plugin searching) i am not sure we want to go further to limit directory depth where plugin can live. it has two drawbacks IMO: 1) it is hard to enforce, not sure everyone will read code/readme. 2) it provides less flexibility to plugins: if want to organize many plugin socket files with directory.
These are definitely good suggestions! It seems like there might be some confusion as to why we are thinking of doing this: Most plugins use the plugin directory to store data (actually a lot of files / dirs) Watching all the directories and files created inside the plugin directory puts more burden on the Kubelet in terms of cpu / memory consumption Watching all the directories and files created inside the plugin directory quickly raises the number of fd consumed By limiting the search of the plugins to one directory, we would be severely limiting the ability of the module to unintentionally harm the overall Kubelet. It seems actually pretty easy to enforce, as it's a simple condition. As for not everyone reading code / readme, we can add the documentation to the gRPC interface if this is a concern. In terms of flexibility, is there a real need to create a plugin in a lower directory? Is there a plugin that has so many sockets that a single directory underneath the plugin directory isn't enough? Naively it seems to me that is enough for the different plugins we enable.
ping on the above :)
Naively it seems to me that $ is enough for the different plugins we enable. i can not speak from plugin developer, but both below looks reasonable to me , or as long as we can communicate to plugin developers clearly and effectively (gRPC interface documentation is needed), the directory depth restriction is reasonable to me.
I like . However, why not put plugintype one level up as . For instance
Is the suggestion to hard code the number of levels that the recursion algorithm will search to 3? Or just expect a socket only at a hard coded level (e.g. socket must be nested 3 deep)? Copying my comment from for consistency: Another option I can think of: recursively scanning but before scanning a directory check if it is mounted (). This will be backwards compatible and will prevent scanning of volume data Thoughts?
i like this idea better if it is always the case that: volume data is mounted, we can modify the watcher code to not watch the mounted directory in the first place.
Skipping mounted dir along with restructuring search path to known (shallow) dir sounds like a good plan. Though with the new suggested paths, sockets will be placed at a location where mounts should not exist.
I think that would be a safe assumption. The plugin directory is only used as a target path for global mounts.
Ephemeral volume types may not need to mount, it could just be a symlink or plain directory. Can we assume any ephemeral volume CSI driver won't implement global mount?
Yes, in-tree ephemeral volumes today don't do a global mount so the mounts directory is which is not the plugin dir. So no issue there. For CSI the will be the global mount path (this plugin dir). Emphermal volumes shouldn't implement call -- but nothing prevents them from doing so. Maybe let's start with just checking for mount and ignoring mount directories. If it is still an issue we can consider adding a configurable black list of paths. Thoughts?
That sounds good. Also, putting the socket files in a specific location that does not clash with mounts will also help (i.e. )
/milestone v1.13 /priority important-soon
I just want to point out that one of the issue we have with recursively watching all directories is that it will significantly increase the amount of fd consumed. Given that plugins actually store data there and in a dynamic way my fear is that it will exhaust the available fds for no real good reasons. Limiting the depth sounds like a reasonable solution to that issue.
btw, because of this verbosity in my cluster writes 5G logs every day and filled up my rootfs (which is small). Is there any way to decrease this verbosity?
We can do both. Don't search mounted dirs, and max depth of search. Who is working on this?
That's pretty bad, can you open another issue to track that.
we have a PR in flight discussing this issue (recursive scanning) here: Thinking more about this, the fact that too much logs is generated is clearly linked to this specific issue and how we process plugins. In any case I'll be fixing that issues too
how confident are we in getting this fixed by Code freeze tomorrow? 5pm PST
I expect things to wind up today, changes in the above PR aren't very contentious and don't involve significant changes

Discussion:
package config const ( <del> DefaultKubeletPodsDirName             = "pods" DefaultKubeletVolumesDirName          = "volumes" DefaultKubeletVolumeDevicesDirName    = "volumeDevices" DefaultKubeletPluginsDirName          = "plugins" DefaultKubeletContainersDirName       = "containers" DefaultKubeletPluginContainersDirName = "plugin-containers" DefaultKubeletPodResourcesDirName     = "pod-resources" </del> <ins> DefaultKubeletPodsDirName                = "pods" DefaultKubeletVolumesDirName             = "volumes" DefaultKubeletVolumeDevicesDirName       = "volumeDevices" DefaultKubeletPluginsDirName             = "plugins" DefaultKubeletPluginsRegistrationDirName = "plugins_registry" DefaultKubeletContainersDirName          = "containers" DefaultKubeletPluginContainersDirName    = "plugin-containers" DefaultKubeletPodResourcesDirName        = "pod-resources" </ins> )

return nil, err } if klet.enablePluginsWatcher { <del> klet.pluginWatcher = pluginwatcher.NewWatcher(klet.getPluginsDir()) </del> <ins> klet.pluginWatcher = pluginwatcher.NewWatcher(klet.getPluginsRegistrationDir()) </ins> } // If the experimentalMounterPathFlag is set, we do not want to

if err := os.MkdirAll(kl.getPluginsDir(), 0750); err != nil { return fmt.Errorf("error creating plugins directory: %v", err) } <ins> if err := os.MkdirAll(kl.getPluginsRegistrationDir(), 0750); err != nil { return fmt.Errorf("error creating plugins registry directory: %v", err) } </ins> if err := os.MkdirAll(kl.getPodResourcesDir(), 0750); err != nil { return fmt.Errorf("error creating podresources directory: %v", err) }

return filepath.Join(kl.getRootDir(), config.DefaultKubeletPluginsDirName) } <ins> // getPluginsRegistrationDir returns the full path to the directory under which // plugins socket should be placed to be registered. // More information is available about plugin registration in the pluginwatcher // module func (kl *Kubelet) getPluginsRegistrationDir() string { return filepath.Join(kl.getRootDir(), config.DefaultKubeletPluginsRegistrationDirName) } </ins> // getPluginDir returns a data directory name for a given plugin name. // Plugins can use these directories to store data that they need to persist. // For per-pod plugin data, see getPodPluginDir.

} // Handle filesystem notify event. <ins> // Files names: // - MUST NOT start with a '.' </ins> func (w *Watcher) handleCreateEvent(event fsnotify.Event) error { klog.V(6).Infof("Handling create event: %v", event)

} if strings.HasPrefix(fi.Name(), ".") { <del> klog.Errorf("Ignoring file: %s", fi.Name()) </del> <ins> klog.V(5).Infof("Ignoring file (starts with '.'): %s", fi.Name()) </ins> return nil } if !fi.IsDir() { <ins> if fi.Mode()&os.ModeSocket == 0 { klog.V(5).Infof("Ignoring non socket file %s", fi.Name()) return nil } </ins> return w.handlePluginRegistration(event.Name) }

plugin, ok := w.getPlugin(event.Name) if !ok { <del> return fmt.Errorf("could not find plugin for deleted file %s", event.Name) </del> <ins> klog.V(5).Infof("could not find plugin for deleted file %s", event.Name) return nil </ins> } // You should not get a Deregister call while registering a plugin

volumes: - name: registration-dir hostPath: <del> path: /var/lib/kubelet/plugins/ </del> <ins> path: /var/lib/kubelet/plugins_registry/ </ins> type: Directory - name: kubelet-dir hostPath:

type: DirectoryOrCreate name: mountpoint-dir - hostPath: <del> path: /var/lib/kubelet/plugins </del> <ins> path: /var/lib/kubelet/plugins_registry </ins> type: Directory name: registration-dir

var _ = framework.KubeDescribe("Device Plugin [Feature:DevicePluginProbe][NodeFeature:DevicePluginProbe][Serial]", func() { f := framework.NewDefaultFramework("device-plugin-errors") <del> testDevicePlugin(f, true, "/var/lib/kubelet/plugins/") </del> <ins> testDevicePlugin(f, true, "/var/lib/kubelet/plugins_registry") </ins> }) func testDevicePlugin(f *framework.Framework, enablePluginWatcher bool, pluginSockDir string) {