Change:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Race conditions found on 2 usages of sync.Cond in client-go/tools/cache.DeltaFIFO and client-go/tools/watch/eventProcessor. I don't know how harmful they are since the buggy interleaving is hard to trigger. They may lead to goroutine leaks to my understanding. They have similar causes and I put them in one issue here for ease of triage. When and are invoked concurrently on an empty queue, the in may miss the signal from . The is guarded by the for-loop below. It appears that should not be called if the FIFO queue is already closed, hence the check at line 475. However, the check above may race with : Specifically, when the queue is empty and not closed, will reach the . Between the check and , another goroutine can call and the broadcast on will be missed. Consequently, may block indefinintely and miss the broadcast on from . Note that holds while and hold . So can interleave with . Here is a stack trace found by our tool while running . The tool is based on the Go race detector. I did some research and found introduced the bug while trying to fix exactly the issue: "stop controller when the stop channel is closed and the queue is empty". The fix doesn't appear to solve the issue completely. One suggested fix: use in and instead of . This would enforce atomicity between the check and in . This is similar to DeltaFIFO. In between and : Another goroutine can call : This would cause the signal on to be missed by the above , leading to the goroutine being blocked unexpectedly. Stack trace from : One suggested fix: use mutex to protect in . What you expected to happen: No race condition. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): MacOS Kernel (e.g. ): Darwin Install tools: Network plugin and version (if this is a network-related bug): Others: This is my first issue filed. Please let me know if I can provide further information.
/sig api-machinery
For issue 1, in probably the date race is not uncovered due to Run() being long running. If we use f.lock in f.Close() and f.IsClosed(), closedLock is not needed any more. Since f.lock is taken at the beginning of Pop(), we would need lockless version of IsClosed()
Oh..Right. I didn't see this point. Please ignore my suggested fix for issue 1 then.
/cc
Hi, I reproduced the goroutine leak with this test case. The control loop inside may not exit when is called on an empty queue due to the race condition. Also, I submitted a PR trying to fix it. :smile:

Discussion:
// Indication the queue is closed. // Used to indicate a queue is closed so a control loop can exit when a queue is empty. // Currently, not used to gate any of CRED operations. <del> closed     bool closedLock sync.Mutex </del> <ins> closed bool </ins> // emitDeltaTypeReplaced is whether to emit the Replaced or Sync // DeltaType when Replace() is called (to preserve backwards compat).

// Close the queue. func (f *DeltaFIFO) Close() { <del> f.closedLock.Lock() defer f.closedLock.Unlock() </del> <ins> f.lock.Lock() defer f.lock.Unlock() </ins> f.closed = true f.cond.Broadcast() }

// IsClosed checks if the queue is closed func (f *DeltaFIFO) IsClosed() bool { <del> f.closedLock.Lock() defer f.closedLock.Unlock() </del> <ins> f.lock.Lock() defer f.lock.Unlock() </ins> return f.closed }

} } } <ins> // TestDeltaFIFO_PopShouldUnblockWhenClosed checks that any blocking Pop on an empty queue // should unblock and return after Close is called. func TestDeltaFIFO_PopShouldUnblockWhenClosed(t *testing.T) { f := NewDeltaFIFOWithOptions(DeltaFIFOOptions{ KeyFunction: testFifoObjectKeyFunc, KnownObjects: literalListerGetter(func() []testFifoObject { return []testFifoObject{mkFifoObj("foo", 5)} }), }) c := make(chan struct{}) const jobs = 10 for i := 0; i < jobs; i++ { go func() { f.Pop(func(obj interface{}) error { return nil }) c <- struct{}{} }() } runtime.Gosched() f.Close() for i := 0; i < jobs; i++ { select { case <-c: case <-time.After(500 * time.Millisecond): t.Fatalf("timed out waiting for Pop to return after Close") } } } </ins>

// Indication the queue is closed. // Used to indicate a queue is closed so a control loop can exit when a queue is empty. // Currently, not used to gate any of CRED operations. <del> closed     bool closedLock sync.Mutex </del> <ins> closed bool </ins> } var (

// Close the queue. func (f *FIFO) Close() { <del> f.closedLock.Lock() defer f.closedLock.Unlock() </del> <ins> f.lock.Lock() defer f.lock.Unlock() </ins> f.closed = true f.cond.Broadcast() }

// IsClosed checks if the queue is closed func (f *FIFO) IsClosed() bool { <del> f.closedLock.Lock() defer f.closedLock.Unlock() </del> <ins> f.lock.Lock() defer f.lock.Unlock() </ins> if f.closed { return true }

// When the queue is empty, invocation of Pop() is blocked until new item is enqueued. // When Close() is called, the f.closed is set and the condition is broadcasted. // Which causes this loop to continue and return from the Pop(). <del> if f.IsClosed() { </del> <ins> if f.closed { </ins> return nil, ErrFIFOClosed }

import ( "fmt" "reflect" <ins> "runtime" </ins> "testing" "time" )

} } } <ins> // TestFIFO_PopShouldUnblockWhenClosed checks that any blocking Pop on an empty queue // should unblock and return after Close is called. func TestFIFO_PopShouldUnblockWhenClosed(t *testing.T) { f := NewFIFO(testFifoObjectKeyFunc) c := make(chan struct{}) const jobs = 10 for i := 0; i < jobs; i++ { go func() { f.Pop(func(obj interface{}) error { return nil }) c <- struct{}{} }() } runtime.Gosched() f.Close() for i := 0; i < jobs; i++ { select { case <-c: case <-time.After(500 * time.Millisecond): t.Fatalf("timed out waiting for Pop to return after Close") } } } </ins>