Change:
/kind feature Here are the possible code cleanup/refactoring we have discussed during device plugin PR code review or offline: Kubelet side device plugin is structured as three-layers: cm.DevicePluginHandler -deviceplugin.Manager -deviceplugin.endpoint. We may improve code readability by dropping them to two: cm.DevicePluginHandler -deviceplugin.Endpoint. Some of the functionalities of deviceplugin.Manager will be moved to cm.DevicePluginHandler and some may be moved to deviceplugin.Endpoint. current deviceplugin.Endpoint has two methods: list and listAndWatch that make the same gRPC call. We may merge them into one. Combined with 1) item, the deviceplugin.Endpoint interface may expose the following methods: Start(), Stop(), Alloc(), run(). Interface cleanup: some of the interfaces in deviceplugin pkg is not necessary public and can be changed to internal. Most functions in doesn't seem to be necessary. May consider to remove this file.
/cc
/sig node
I would work on this. Will submit a PR later.
...actually, I'm working on it too, so have you started yet?
Yes. Already started.
Okey, I'm concentrating on part of refactoring
I think there is no need to split this into several PRs. I'll cover it all in . Thanks.
Well, in fact, I have finished main part of refactoring
We should avoid doing duplicate work. Better notify others before starting working. Since you've finished the main part of refactoring, I will close my pr.
I'm so sorry, I discussed with jiayingz in slack but forgot to response in the issue itself
Hi ! Sorry to be commenting so late on this. I'll start with a short list of the Exposed interfaces / Functions of the deviceplugin module: Manager: , , , , , , Endpoint: Noting exposed Device Plugin Stub: , , , , Types: , Utils: , Not sure what interfaces you are referring to. Looking at the interfaces exposed, and might be very similar though. But this seem to be mostly it, am I missing something ? This might be a different conception or code model that we have. I do believe in separation of concerns and decoupling and the cleanest code I've read out there have been applying these. You seem to be suggesting otherwise here by trying to merge the gRPC server (manager) in the . I already feel like the is very bloated and assumes multiple roles I'm not sure if adding one more role to it is a good idea. IMHO the current layout in the device plugin seems readable enough and follows a pretty good separation of concerns. I don't feel the same of the where functions seem to be assuming more than one role. I tried to wrap my head around this problem and didn't find any acceptable solutions :( If someone can find a solution to this that would be great! As mentioned, might be merged with as they seem very similar :) However I don't see the value in increasing cognitive load on the dev reading the code by adding utils functions in the "Business Logic Code". This seem to go against the stated objective to "improve code readability".
Before anyone posts a PR, let's first settle down on the right abstractions and problems to be solved. So a discussion and possibly a rough prototype will help avoid tedious code reviews.
So I think the key point is if it necessary to combine the part of Manager into DevicePluginHandler and endpoint. From a design perspective, it seems clearer to keep it independent; However, people might feel it redundant when looking into the code, since the manager itself does not have much logic, largely works as passthrough, except for the gRPC server part. As for the merge of func listAndWatch & list. , I think jiayingz means that since we won't call the list func periodically as we do in the mean list/watch in k8s, it's basically a premise for func listAndWatch here, so we may consider to combine them into one
Is it possible that you break your change into small PRs? Possible one PR for a clean-up item mentioned in the issue. This way, it may be easier to settle down and also makes review faster.
Sure. But as said, shall we have a discussion about direction of the refactoring first? First seeing the issue, I thought you major members of the feature already had a discussion offline, but it seems that Renaud still have doubts about the idea of breaking Manager into DevicePluginHandler and endpoint. After we reach an agreement. I would set about the modification at once :)
I think the overall goals of this issue is to improve readability and maintainability of the device plugin through certain code refactoring. The listed items are possibilities we may consider and reflects the general concerns that the reviewers had during the device plugin PR review. We do hope whoever takes this task would put more thoughts on them and come up with an implementation that helps achieve the overall goals. In terms of reaching overall agreements on the refactoring, it sometimes could take quite a lot communication overhead. I think it is also why suggested that a rough prototype could help avoid tedious code reviews. The prototype however needs to reflect the overall goals of improving readability and maintainability (unit test coverage is part of this and shouldn't be sacrificed). If someone is seeking more specific implementation tasks, I would suggest to start with smaller items, like cleaning up functions in
Thanks for explanation, I would break the PR into small ones according to optional refactoring items :)
As discussed, the change has been broken into small PRs as follows: Deviceplugin refactoring: cleanup some unnecessary functions Deviceplugin refactoring: merge func list and listwatch in endpoint into one Deviceplugin refactoring: break Manager into DevicePluginHandler and endpoint
Hi! Sorry for the late answers... Right, as you mentioned, it's in charge of handling and the gRPC server. Furthermore it exposes a clean, readable and logical interface. Which seems reason enough for me to keep it as a separate module as it would not make sense to expose this interface elsewhere. oh this is what you meant! This was the shape of the original function. However IMHO shoving code in one big function does not seem like a good programming model and seems to hamper readability. Furthermore it made more sense to me to separate both tasks as one is listing the devices and the other is watching them. We can try to have list and watch call that function at the start. I don't have a strong opinion on moving the call. However not sure it really brings any value to merge the functions :) I think it's a good vision! I would suggest a few objectives: Separate the responsibilities currently handled by the devicepluginhandler before 1.8 release. A few tasks might be (though I'd like your opinion on this): should it be in it's own separate file ? shouldn't this be in the file ? should the functions be broken up in multiple ones (/files) that only accomplish one thing: Resource management Device Selection Checkpointing (does it make sense to define an interface for it ?) Merge the redundant function in from before 1.8 (I don't understand why people want to remove this file ?) Make a utils function as it does not need to be a method of the Remove check as the socket is removed just before Increase the test coverage of devicepluginhandler, manager and endpoint to at least 80% Add e2e tests to handle the following cases: Normal allocation Kubelet failure Device failure Both failures Can you point to these comments? I just went through your PR and could not find these.
, Renaud Gaubert <: dropping to two layers is to simplify the code, especially locking. it in which I think should only contain public interfaces. am not a big fan of having too many small files. After we move podDevices functions out, the file is only about 200 line of code, which I think is at reasonable size. here should be for 1.9. and issue for e2e tests. Jiaying
Thanks for the comment. Most of the objectives you put forward sound reasonable. In fact some of the them have been contained in existing PRs, say Merge the redundant function IsDeviceName in IsResourceNameValid from before 1.8 & Since some of us still have doubts about the idea of and , I will extract refactoring options for sure to separate PRs, and keep PR for further discussion
Hi ! dropping to two layers is to simplify the code, especially locking. In the existing PRs, I don't see a code simplification. Actually IMHO dropping an interface is more the reverse of improving readability. I'd rather we keep the code separated, which achieves at least separation of concern and maintainable code on which we can build. Readability is an objective but merging the layers is not achieving this and comes at the cost of removing separation of concerns and maintainable code. This is not coding style but rather basic code design principles (single responsibility principle). That makes a lot of sense! File size and coding style is not the point. I'm talking of separation of concerns, not number of lines. Having modular code that does one thing seem like a reasonable goal in terms of readability and code structure. If we want to continue adding features and maintain this code, everyone will be much more confortable knowing that changes they make in one part of the code will not impact other parts of the code. This is especially true if we continue to converge towards this one giant structure handling everything This argument is the same for the 2-3 layer structure. If it means that we need an additional mutex, I'm fine with that, it hardly impacts performance but on the other hand it keeps the code modular and separates concerns.
/reopen
you can't re-open an issue/PR unless you authored it or you are assigned to it. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/reopen
/unassign
/assign
GitHub didn't allow me to assign the following users: lichuqiang. Note that only can be assigned. <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign

Discussion:
} // newEndpoint creates a new endpoint for the given resourceName. <del> func newEndpoint(socketPath, resourceName string, callback MonitorCallback) (*endpoint, error) { </del> <ins> func newEndpoint(socketPath, resourceName string, devices map[string]pluginapi.Device, callback MonitorCallback) (*endpoint, error) { </ins> client, c, err := dial(socketPath) if err != nil { glog.Errorf("Can't create new endpoint with path %s err %v", socketPath, err)

socketPath:   socketPath, resourceName: resourceName, <del> devices:  nil, </del> <ins> devices:  devices, </ins> callback: callback, }, nil }

return devs } <del> // list initializes ListAndWatch gRPC call for the device plugin and gets the // initial list of the devices. Returns ListAndWatch gRPC stream on success. func (e *endpoint) list() (pluginapi.DevicePlugin_ListAndWatchClient, error) { stream, err := e.client.ListAndWatch(context.Background(), &pluginapi.Empty{}) if err != nil { glog.Errorf(errListAndWatch, e.resourceName, err) return nil, err } devs, err := stream.Recv() if err != nil { glog.Errorf(errListAndWatch, e.resourceName, err) return nil, err } devices := make(map[string]pluginapi.Device) var added, updated, deleted []pluginapi.Device for _, d := range devs.Devices { devices[d.ID] = *d added = append(added, *d) } e.mutex.Lock() e.devices = devices e.mutex.Unlock() e.callback(e.resourceName, added, updated, deleted) return stream, nil } // listAndWatch blocks on receiving ListAndWatch gRPC stream updates. Each ListAndWatch </del> <ins> // run initializes ListAndWatch gRPC call for the device plugin and // blocks on receiving ListAndWatch gRPC stream updates. Each ListAndWatch </ins> // stream update contains a new list of device states. listAndWatch compares the new // device states with its cached states to get list of new, updated, and deleted devices. // It then issues a callback to pass this information to the device_plugin_handler which // will adjust the resource available information accordingly. <del> func (e *endpoint) listAndWatch(stream pluginapi.DevicePlugin_ListAndWatchClient) { </del> <ins> func (e *endpoint) run() { </ins> glog.V(3).Infof("Starting ListAndWatch") <ins> stream, err := e.client.ListAndWatch(context.Background(), &pluginapi.Empty{}) if err != nil { glog.Errorf(errListAndWatch, e.resourceName, err) return } </ins> devices := make(map[string]pluginapi.Device) e.mutex.Lock()

defer ecleanup(t, p, e) } <del> func TestList(t *testing.T) { socket := path.Join("/tmp", esocketName) devs := []*pluginapi.Device{ {ID: "ADeviceId", Health: pluginapi.Healthy}, } p, e := esetup(t, devs, socket, "mock", func(n string, a, u, r []pluginapi.Device) {}) defer ecleanup(t, p, e) _, err := e.list() require.NoError(t, err) e.mutex.Lock() defer e.mutex.Unlock() require.Len(t, e.devices, 1) d, ok := e.devices[devs[0].ID] require.True(t, ok) require.Equal(t, d.ID, devs[0].ID) require.Equal(t, d.Health, devs[0].Health) } func TestListAndWatch(t *testing.T) { </del> <ins> func TestRun(t *testing.T) { </ins> socket := path.Join("/tmp", esocketName) devs := []*pluginapi.Device{

}) defer ecleanup(t, p, e) <del> s, err := e.list() require.NoError(t, err) go e.listAndWatch(s) </del> <ins> go e.run() </ins> p.Update(updated) time.Sleep(time.Second)

err := p.Start() require.NoError(t, err) <del> e, err := newEndpoint(socket, "mock", func(n string, a, u, r []pluginapi.Device) {}) </del> <ins> e, err := newEndpoint(socket, "mock", make(map[string]pluginapi.Device), func(n string, a, u, r []pluginapi.Device) {}) </ins> require.NoError(t, err) return p, e

} const ( <del> // defaultCheckpoint is the file name of device plugin checkpoint defaultCheckpoint = "kubelet_internal_checkpoint" </del> <ins> // kubeletDevicePluginCheckpoint is the file name of device plugin checkpoint kubeletDevicePluginCheckpoint = "kubelet_internal_checkpoint" </ins> ) // CheckpointFile returns device plugin checkpoint file path. func (m *ManagerImpl) CheckpointFile() string { <del> return filepath.Join(m.socketdir, defaultCheckpoint) </del> <ins> return filepath.Join(m.socketdir, kubeletDevicePluginCheckpoint) </ins> } // Start starts the Device Plugin Manager

} func (m *ManagerImpl) addEndpoint(r *pluginapi.RegisterRequest) { <ins> existingDevs := make(map[string]pluginapi.Device) m.mutex.Lock() old, ok := m.endpoints[r.ResourceName] if ok && old != nil { // Pass devices of previous endpoint into re-registered one, // to avoid potential orphaned devices upon re-registration existingDevs = old.devices } m.mutex.Unlock() </ins> socketPath := filepath.Join(m.socketdir, r.Endpoint) <del> e, err := newEndpoint(socketPath, r.ResourceName, m.callback) </del> <ins> e, err := newEndpoint(socketPath, r.ResourceName, existingDevs, m.callback) </ins> if err != nil { glog.Errorf("Failed to dial device plugin with request %v: %v", r, err) return } <del> stream, err := e.list() if err != nil { glog.Errorf("Failed to List devices for plugin %v: %v", r.ResourceName, err) </del> <ins> m.mutex.Lock() // Check for potential re-registration during the initialization of new endpoint, // and skip updating if re-registration happens. // TODO: simplify the part once we have a better way to handle registered devices ext := m.endpoints[r.ResourceName] if ext != old { glog.Warningf("Some other endpoint %v is added while endpoint %v is initialized", ext, e) m.mutex.Unlock() </ins> e.stop() return } // Associates the newly created endpoint with the corresponding resource name. // Stops existing endpoint if there is any. <del> m.mutex.Lock() old, ok := m.endpoints[r.ResourceName] </del> m.endpoints[r.ResourceName] = e glog.V(2).Infof("Registered endpoint %v", e) m.mutex.Unlock() <del> if ok && old != nil { </del> <ins> if old != nil { </ins> old.stop() } go func() { <del> e.listAndWatch(stream) </del> <ins> e.run() </ins> e.stop() m.mutex.Lock()

// Tests that the device plugin manager correctly handles registration and re-registration by // making sure that after registration, devices are correctly updated and if a re-registration <del> // happens, we will NOT delete devices. </del> <ins> // happens, we will NOT delete devices; and no orphaned devices left. </ins> func TestDevicePluginReRegistration(t *testing.T) { devs := []*pluginapi.Device{ {ID: "Dev1", Health: pluginapi.Healthy}, {ID: "Dev2", Health: pluginapi.Healthy}, } <ins> devsForRegistration := []*pluginapi.Device{ {ID: "Dev3", Health: pluginapi.Healthy}, } </ins> callbackCount := 0 callbackChan := make(chan int) var stopping int32 stopping = 0 callback := func(n string, a, u, r []pluginapi.Device) { <del> // Should be called twice, one for each plugin registration, till we are stopping. if callbackCount > 1 && atomic.LoadInt32(&stopping) <= 0 { </del> <ins> // Should be called three times, one for each plugin registration, till we are stopping. if callbackCount > 2 && atomic.LoadInt32(&stopping) <= 0 { </ins> t.FailNow() } callbackCount++

devices2 := m.Devices() require.Equal(t, 2, len(devices2[testResourceName]), "Devices shouldn't change.") <ins> // Test the scenario that a plugin re-registers with different devices. p3 := NewDevicePluginStub(devsForRegistration, pluginSocketName+".third") err = p3.Start() require.NoError(t, err) p3.Register(socketName, testResourceName) // Wait for the second callback to be issued. <-callbackChan devices3 := m.Devices() require.Equal(t, 1, len(devices3[testResourceName]), "Devices of plugin previously registered should be removed.") </ins> // Wait long enough to catch unexpected callbacks. time.Sleep(5 * time.Second) atomic.StoreInt32(&stopping, 1) <del> cleanup(t, m, p1) </del> p2.Stop() <ins> p3.Stop() cleanup(t, m, p1) </ins> } func setup(t *testing.T, devs []*pluginapi.Device, callback MonitorCallback) (Manager, *Stub) {