Change:
While Debugging Specifically with the unit test which is part of the endpoint controller's tests I ran into the following race after adding some code to shutdown the following conclusion of the unit tests And after digging a little deeper I found the problem in the function, specifically a recorder may try to send an event to a broadcaster's incoming event channel even if the Broadcaster is shutdown Adding the following check to ensure we drop the event if the Broadcaster is already dropped seems to fix things. I will push a PR to fix this soon.
/transfer kubernetes
/sig api-machinery
/cc (since you are all commenting in the respective PR) Thanks! /triage accepted
/assign he already has a PR and David's blessing
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
should address this once merged.

Discussion:
distributing sync.WaitGroup // incomingBlock allows us to ensure we don't race and end up sending events <del> // to a closed channel following a brodcaster shutdown. </del> <ins> // to a closed channel following a broadcaster shutdown. </ins> incomingBlock sync.Mutex incoming      chan Event stopped       chan struct{}

// won't ever see that event, and will always see any event after they are // added. func (m *Broadcaster) blockQueue(f func()) { <ins> m.incomingBlock.Lock() defer m.incomingBlock.Unlock() </ins> select { case <-m.stopped: return

// have received the data yet as it can remain sitting in the buffered // channel. It will block until the broadcaster stop request is actually executed func (m *Broadcaster) Shutdown() { <del> m.incomingBlock.Lock() defer m.incomingBlock.Unlock() </del> m.blockQueue(func() { close(m.stopped) close(m.incoming)

limitations under the License. */ <del> package watch_test </del> <ins> package watch </ins> import ( "reflect"

"k8s.io/apimachinery/pkg/runtime" "k8s.io/apimachinery/pkg/runtime/schema" "k8s.io/apimachinery/pkg/util/wait" <del> . "k8s.io/apimachinery/pkg/watch" </del> ) type myType struct {

assert.Equal(t, sendOnClosed, false, "ActionOrDrop should return false if broadcaster is already shutdown") assert.EqualError(t, err, "broadcaster already stopped", "ActionOrDrop should report error id broadcaster is shutdown") } <ins> // Test this since we see usage patterns where the broadcaster and watchers are // stopped simultaneously leading to races. func TestBroadcasterShutdownRace(t *testing.T) { m := NewBroadcaster(1, WaitIfChannelFull) stopCh := make(chan struct{}) // Add a bunch of watchers const testWatchers = 2 for i := 0; i < testWatchers; i++ { i := i _, err := m.Watch() if err != nil { t.Fatalf("Unable start event watcher: '%v' (will not retry!)", err) } // This is how we force the watchers to close down independently of the // eventbroadcaster, see real usage pattern in startRecordingEvents() go func() { <-stopCh t.Log("Stopping Watchers") m.stopWatching(int64(i)) }() } event := Event{Type: Added, Object: &myType{"foo", "hello world"}} err := m.Action(event.Type, event.Object) if err != nil { t.Fatalf("error sending event: %v", err) } // Manually simulate m.Shutdown() but change it to force a race scenario // 1. Close watcher stopchannel, so watchers are closed independently of the // eventBroadcaster // 2. Shutdown the m.incoming slightly Before m.stopped so that the watcher's // call of Blockqueue can pass the m.stopped check. m.blockQueue(func() { close(stopCh) close(m.incoming) time.Sleep(1 * time.Millisecond) close(m.stopped) }) m.distributing.Wait() } </ins>