Change:
} e.mutex.Lock() <del> e.devices = devices </del> <ins> // NOTE: Return a copy of 'devices' instead of returning a direct reference to local 'devices' e.devices = make(map[string]pluginapi.Device) for _, d := range devices { e.devices[d.ID] = d } </ins> e.mutex.Unlock() e.callback(e.resourceName, added, updated, deleted)

Discussion:
Is this a BUG REPORT or FEATURE REQUEST?: /kind bug /area hw-accelerators /sig node What happened: When re-writing the endpoint code I discovered that the is wrong and can lead to a data race while calling the function. The current logic is a bit convoluted and would make more sense if the map was created in the for loop. However it seems like once the map is assigned to the structure, a device update while calling the function will lead to a data race, Anything else we need to know?: /assign
Not quite sure about the race scenario you described, which func do you mean by function ? The ?
Did you mean by ?  map is being assigned to structure with locking protection only.  Did you want to say that a race will happen if is being updated and at the same time gets invoked? If yes, i dont think so. At both the places, first lock is taken. So only one can run at a time and other will wait on mutex. Please let me know if i mistaken something.
Sorry I meant  though essentially the manager.Devices function calls the endpoint.getDevices(). Yes thanks for clarifying this.
I believe we all agree that there is no race in the scenario described in the description of this issue. Shall we close this issue?
there is a race, I think I didn't fully read your answer at the time sorry. Because the way maps works when the map in the structure, It is then making the Endpoint's map point to the local copy of . Which means that every time we lock in the that and thus the endpoint's map .
thanks! I got confused b/w call-by-value and call-by-reference and forgot that in golang maps are call by reference.
this possibility still remains that some devices may disappear from the endpoint, while manager knows about those.
Yes. That is the reason why in the latest commit in the manager is taking a snapshot of the devices known and based the device allocation on that. There are possibilities where devices come and go between the two points in time: See above diagram. At time the manager starts processing the device allocation requests for a Pod. It takes a snapshot of the all devices it knows. It doesn't care how many great devices appeared in history or how many devices may go dead in seconds.

The manager cannot predict that, so "let it be" is its attitude towards device changes. At time , it tries allocate device ID . So long is good at the moment, it proceeds to allocate and . If and only when all requested devices can be allocated, the Pod can be successfully created. It could be the case that device A3 has gone offline after and then come back to life soon before . That is acceptable. As a transaction, the Pod device allocation is okay. We cannot pretend that we can get well prepared for all device failures. For example, after the manager has confirmed the device allocation, may fail again. The Pod still cannot use it. But there is nothing we can do (it is life...).
The race this issue is discussing is about a "real" race where mutex locks are not handled correctly. The issue you are mentioning is a different one that explains very well in the above comment. It's a complex interaction but is currently correctly handled by the allocation and pod rejection process.