Change:
Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: I'm running into a test failure during "make test". To get more output and to avoid issue , I am using: What you expected to happen: The test should pass. How to reproduce it (as minimally and precisely as possible): It happens for me only when running all tests, but then it happens reliably. When limiting the test run with , it works. The failing calls are not surprising, because got unlinked after binding to it. Those might come from 's : it removes everything in the socket dir, which happens to be . For comparison, here's a successful run: Anything else we need to know?: I'm wondering whether it is guaranteed that different tests in the same .go file run serialized. Does anyone know for sure? I'm asking because all tests in use the same socket directory and thus would conflict with each other if they were to run in parallel. Note that the and calls in the failed test are for different pids. I've not checked whether those are truly different processes (as in fork+exec) or worker threads (clone). Environment: Kubernetes version (use ): master branch, commit OS (e.g. from /etc/os-release): Debian Stretch (9.0) Kernel (e.g. ): 4.9.65-3+deb9u2
After fixing the /tmp dir handling with PR I can no longer reproduce this issue. It might be worth keeping this issue open nevertheless, because the issue has not been explained completely and there might be some remaining problem lurking somewhere.
I think you originally wrote this test. Perhaps you can comment here and comment on PR ?
/cc This test has been re-written multiple times, I'll take a look ASAP :)
/sig node
Will take a look ASAP.
I experienced the same. What's interesting is I can get it to pass when running the devicemanager suite, but only about half the time.
/area hw-accelerators
Thanks a lot for reporting the issue! I think your analysis is right. Looks like the problem is because the devicemanager unit tests are interfering with each other. I will make a change to have each unit test in the file to use a separate test tmp dir.
I'm not exactly sure why, though. Do they really run in parallel? According to my understanding of testing.Parallel, one has to explicitly request parallel execution, and the tests don't do that. When they run sequentially, then it isn't obvious why the tests interfere and thus using per-test socket dirs would be just a band-aid. Please have a look at PR , it has such a change.
Ah sorry I didn't realize you were already working on a fix. I think go test by default run tests in parallel, so we need to make sure each unit test is self contained. That also explains why you hit this problem more easily when running more tests. I think we perhaps need to modify setup() to make sure each newly created manager uses a separate test dir. Maybe you can take a look at my WIP change and see whether you can merge some of its changes into PR ?
I suggest you take a look at the documentation: On Fri, 9 Feb 2018, 22:34 Jiaying Zhang, <:
On Sat, 2018-02-10 at 05:31 +0000, Vikas Choudhary (vikasc) wrote: True, but which unit tests run concurrently? To answer that, I re-read the "testing" documentation and ran a quick experiment. That confirmed that individual tests inside the same test binary run sequentially. A test function has to explicitly call t.Parallel() before the next test starts, and that test blocks unless it also calls t.Parallel(). In other words, using the same /tmp/deviceplugin between all test functions should have been okay. I suspect it goes wrong sometimes because one test function starts a goroutine, does not wait for its completion, and that goroutine then interferes with the next test function when using the same dir, depending on timing. Running other test binaries in parallel and thus increasing system load probably also matters, together with overall number of CPUs. I think merging PR makes sense because a dynamic test directory is more flexible and enables running the tests in parallel (which could be a separate PR or be done as part of PR ), but I also think that it would be good if someone with a better understanding of the code that gets tested here could look into this race condition. [(quite) a bit later...] I chased down the spurious unlinkat() to UnixStream.Close for the socket, which unlinks the socket path [1]. By wrapping the listen socket such that a custom Close logs the call and a stack trace, I now have the smoking gun. [1] With some extra logging one can see that the that was created for TestNewManagerImplStart gets closed and unlinked while TestDevicePluginReRegistration already runs. The goroutine which runs too long is the from was called, but didn't wait for the goroutine to actually stop: That now begs the question: what is the right way to stop a grpc server? Below is the change I made to to track when it closes the socket:
On Tue, 2018-02-13 at 15:57 +0100, Patrick Ohly wrote: Server.Stop is the right way, it just doesn't fully stop the server. I consider this an issue in gRPC, therefore I have created https://github .com/grpc/grpc-go/issues/1861
Is it possible that defer statements are executed at the same time another test might start? EDIT: NVM looks like that's not the case
On Tue, 2018-02-13 at 07:39 -0800, Renaud Gaubert wrote: No, defer statements are guaranteed to run before a function returns. I think they can even change the return code and/or catch a panic.
thanks so much for your detailed debugging analysis and details about the tools you used (e.g. using filtered strace). I'm fairly new to golang and I've been messing around with the k8s source locally trying to get a feel for the project and work towards contributing, and I was debugging this test on my own but flailing against how to instrument the running tests. very much appreciated and nice work!

Discussion:
"net" "os" "path" <ins> "sync" </ins> "time" "google.golang.org/grpc"

socket string stop   chan interface{} <ins> wg     sync.WaitGroup </ins> update chan []*pluginapi.Device server *grpc.Server

m.allocFunc = f } <del> // Start starts the gRPC server of the device plugin </del> <ins> // Start starts the gRPC server of the device plugin. Can only // be called once. </ins> func (m *Stub) Start() error { err := m.cleanup() if err != nil {

return err } <ins> m.wg.Add(1) </ins> m.server = grpc.NewServer([]grpc.ServerOption{}...) pluginapi.RegisterDevicePluginServer(m.server, m) <del> go m.server.Serve(sock) </del> <ins> go func() { defer m.wg.Done() m.server.Serve(sock) }() </ins> _, conn, err := dial(m.socket) if err != nil { return err

return nil } <del> // Stop stops the gRPC server </del> <ins> // Stop stops the gRPC server. Can be called without a prior Start // and more than once. Not safe to be called concurrently by different // goroutines! </ins> func (m *Stub) Stop() error { <ins> if m.server == nil { return nil } </ins> m.server.Stop() <del> close(m.stop) </del> <ins> m.wg.Wait() m.server = nil close(m.stop) // This prevents re-starting the server. </ins> return m.cleanup() }

mutex     sync.Mutex server *grpc.Server <ins> wg     sync.WaitGroup </ins> // activePods is a method for listing active pods on the node // so the amount of pluginResources requested by existing pods

return err } <ins> m.wg.Add(1) </ins> m.server = grpc.NewServer([]grpc.ServerOption{}...) pluginapi.RegisterRegistrationServer(m.server, m) <del> go m.server.Serve(s) </del> <ins> go func() { defer m.wg.Done() m.server.Serve(s) }() </ins> glog.V(2).Infof("Serving device plugin registration server on %q", socketPath)

} // Stop is the function that can stop the gRPC server. <ins> // Can be called concurrently, more than once, and is safe to call // without a prior Start. </ins> func (m *ManagerImpl) Stop() error { m.mutex.Lock() defer m.mutex.Unlock()

e.stop() } <ins> if m.server == nil { return nil } </ins> m.server.Stop() <ins> m.wg.Wait() m.server = nil </ins> return nil }

defer os.RemoveAll(socketDir) m, p := setup(t, []*pluginapi.Device{}, func(n string, a, u, r []pluginapi.Device) {}, socketName, pluginSocketName) cleanup(t, m, p) <ins> // Stop should tolerate being called more than once. cleanup(t, m, p) } func TestNewManagerImplStop(t *testing.T) { socketDir, socketName, pluginSocketName, err := tmpSocketDir() require.NoError(t, err) defer os.RemoveAll(socketDir) m, err := newManagerImpl(socketName) require.NoError(t, err) // No prior Start, but that should be okay. err = m.Stop() require.NoError(t, err) devs := []*pluginapi.Device{ {ID: "Dev1", Health: pluginapi.Healthy}, {ID: "Dev2", Health: pluginapi.Healthy}, } p := NewDevicePluginStub(devs, pluginSocketName) // Same here. err = p.Stop() require.NoError(t, err) </ins> } // Tests that the device plugin manager correctly handles registration and re-registration by