Change:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug /sig node What happened: i register device by my deviceplugin. It works good. Then i stopmy plugin, k8s mark resources Unhealthy as expect. But when i restart plugin. The resource does not recover, remain allocatable 0. What you expected to happen: Resource turns healthy when plugin restart. How to reproduce it (as minimally and precisely as possible): k8s device plugin device plugin device plugin Anything else we need to know?: Environment: Kubernetes version (use ): v1.10.1 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Ubuntu Kernel (e.g. ): Install tools: Others:
devicemanger set devices unhealthy when socket closed. But devices in endpoint, devices remain health, and will not call callback function when watched new device request (due to same health), and devicemanger' devices will never recover.
I was able to reproduce this. would this be a proper fix or we need something more sophisticated?
May be less destructive variant would be better?
/area hw-accelerators /cc
I thought when the new device plugin re-registers, we would create a new endpoint.
Ah I think I see your point. The fix you proposed in sounds reasonable, but we may need to expose a function from with proper locking.
Btw, when we are fixing this, I think we should extend the existing test/e2enode/[gpu] to verify the fix by restarting the device plugin and waiting for resource allocatable change. We once had that test coverage in test/e2e/scheduling/nvidia- but somehow lost it through some code changes.
/cc
[Posting offline question] Why is state stored both in endpoints and device manager?
I have a vague memory that we had some similar discussion before and am trying to remember the reason why the code was structured this way. I think one reason is perhaps that we want to prevent a single buggy device plugin from DOS device manager with flooding ListAndWatch updates by having its endpoint filter out unnecessary updates. do you remember any other reasons?
Endpoints stores device state whereas manager stores "API server" state. These two states are very similar but different. One always has the latest device state. The other is in charge of reporting what devices changed to an unhealthy state and to delete the resource after a certain set of time. So we're not really storing the state twice :)
It might be simpler to make endpoints lean and store all state within the device manager to avoid synchronisation issues. Renaud Gaubert <:
I doubt that any of the fix that mentioned will work. Problem is this -- devices are being copied with health as "Healthy" from old endpoint to new endpoint, And then marking those devices unhealthy in device manager. So what was saying is correct that we create a new endpoint and in the fix , we should update devices health in new endpoint. This can be done at two places-- 1. before copying devices to new endpoint or 2. in the goroutine after e.stop() as is suggesting. I would prefer option 1 because in option2 there could be a race if device health gets updated to "unhealthy" after got invoked in . I will post a fix soon today.
/assign
will handle e2e in a separate PR shortly.
device manager to avoid synchronisation issues. then grpc calls will have to be protected with mutex.
Your fix looks quite implicit in my opinion. It sets device health to Unhealthy in addEndPoint, which is called from Register. This might look confusing. It looks like we make new endpoint unhealthy when we register new resource. My fix synchronises device manager and endpoint health exactly when resource is set to unhealthy state. The code is covered by mutex, so I don't see why it would race. Even if it would I'd argue that your fix could cause future confusions and should be reworked/ thanks for the suggestion to move this functionality into I agree with this. Also calling it from markResourceUnhealthy may look better than my sketchy proposal. Thoughts?
I tested my fix before posting here. It worked. As for your comment - devices are copied from old to new endpoint with the old state: and then passed to newEndpointImpl as existingDevs variable.
My guess is that you might not have hit the race condition in your local testing. If devices are marked unhealthy from within goroutine, like you suggested, this will race against the thread which will invoke for the new endpoint. And if first thread got late in marking devices unhealthy, callback from the will not be invoked. mutex that you talked about is not protecting from the race i described. This mutex is protecting device manager database only. Though i dont think it is confusing to mark devices from old endpoint to be unhealthy because anyways old endpoint is going to be deleted and new endpoint may really not be having devices in healthy state. But still if there is any confusion may be we can extend existing comment smthng like this:
See my comment on your PR :
Is this fixed in 1.10 and/or 1.11?
I believe 1.9.5, 1.9.6, 1.10.0 1.10.1, 1.10.2 are the only releases affected by this bug.
Thanks

Discussion:
// to avoid potential orphaned devices upon re-registration devices := make(map[string]pluginapi.Device) for _, device := range old.getDevices() { <ins> device.Health = pluginapi.Unhealthy </ins> devices[device.ID] = device } existingDevs = devices