Change:
The standard event recorder, when recording an event, calls the underlying Broadcaster in a goroutine, so that even recording is non-blocking. Things like leader election emit events over the course of their lifecycle via this broadcaster. Consider the following scenario: GR1: spawn leader election ----GR2 GR2: Leader election: emit event "Lease Acquired" ----spawn GR3 () GR1: Shutdown leader election ----blocks until GR2 is done GR1: Shutdown broadcaster --() GR3: ----send on closed channel () Practically, we've observed this behavior popping up occaisionally in tests in controller-runtime:
/sig api-machinery /kind bug
<details<summaryrelevant race detector output/stack traces</summary</details
were you planning on sending a fix, or need our help to send one to fix it?
I can probably try to figure out a fix if you want, but the whole model for how event broadcasters currently work is inverse to how Go suggests you manage channels (only writers should ever close channels, not readers). I'll need to do a bit of playing around with things.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
cc
I think this is fixed by . Please reopen if it's not the case.
/triage accepted
Would be possible to cherry-pick this fix in 1.20?

Discussion:
return m } <ins> // NewLongQueueBroadcaster functions nearly identically to NewBroadcaster, // except that the incoming queue is the same size as the outgoing queues // (specified by queueLength). func NewLongQueueBroadcaster(queueLength int, fullChannelBehavior FullChannelBehavior) *Broadcaster { m := &Broadcaster{ watchers:            map[int64]*broadcasterWatcher{}, incoming:            make(chan Event, queueLength), stopped:             make(chan struct{}), watchQueueLength:    queueLength, fullChannelBehavior: fullChannelBehavior, } m.distributing.Add(1) go m.loop() return m } </ins> const internalRunFunctionMarker = "internal-do-function" // a function type we can shoehorn into the queue.

m.incoming <- Event{action, obj} } <ins> // Action distributes the given event among all watchers, or drops it on the floor // if too many incoming actions are queued up.  Returns true if the action was sent, // false if dropped. func (m *Broadcaster) ActionOrDrop(action EventType, obj runtime.Object) bool { select { case m.incoming <- Event{action, obj}: return true default: return false } } </ins> // Shutdown disconnects all watchers (but any queued events will still be distributed). // You must not call Action or Watch* after calling Shutdown. This call blocks // until all events have been distributed through the outbound channels. Note

// Creates a new event broadcaster. func NewBroadcaster() EventBroadcaster { return &eventBroadcasterImpl{ <del> Broadcaster:   watch.NewBroadcaster(maxQueuedEvents, watch.DropIfChannelFull), </del> <ins> Broadcaster:   watch.NewLongQueueBroadcaster(maxQueuedEvents, watch.DropIfChannelFull), </ins> sleepDuration: defaultSleepDuration, } } func NewBroadcasterForTests(sleepDuration time.Duration) EventBroadcaster { return &eventBroadcasterImpl{ <del> Broadcaster:   watch.NewBroadcaster(maxQueuedEvents, watch.DropIfChannelFull), </del> <ins> Broadcaster:   watch.NewLongQueueBroadcaster(maxQueuedEvents, watch.DropIfChannelFull), </ins> sleepDuration: sleepDuration, } } func NewBroadcasterWithCorrelatorOptions(options CorrelatorOptions) EventBroadcaster { return &eventBroadcasterImpl{ <del> Broadcaster:   watch.NewBroadcaster(maxQueuedEvents, watch.DropIfChannelFull), </del> <ins> Broadcaster:   watch.NewLongQueueBroadcaster(maxQueuedEvents, watch.DropIfChannelFull), </ins> sleepDuration: defaultSleepDuration, options:       options, }

event := recorder.makeEvent(ref, annotations, eventtype, reason, message) event.Source = recorder.source <del> go func() { // NOTE: events should be a non-blocking operation defer utilruntime.HandleCrash() recorder.Action(watch.Added, event) }() </del> <ins> // NOTE: events should be a non-blocking operation, but we also need to not // put this in a goroutine, otherwise we'll race to write to a closed channel // when we go to shut down this broadcaster.  Just drop events if we get overloaded, // and log an error if that happens (we've configured the broadcaster to drop // outgoing events anyway). if sent := recorder.ActionOrDrop(watch.Added, event); !sent { klog.Errorf("unable to record event: too many queued events, dropped event %#v", event) } </ins> } func (recorder *recorderImpl) Event(object runtime.Object, eventtype, reason, message string) {

"fmt" "net/http" "strconv" <ins> "sync" </ins> "testing" "time"

} } <ins> func TestNonRacyShutdown(t *testing.T) { // Attempt to simulate previously racy conditions, and ensure that no race // occurs: Nominally, calling "Eventf" *followed by* shutdown from the same // thread should be a safe operation, but it's not if we launch recorder.Action // in a goroutine. caster := NewBroadcasterForTests(0) clock := clock.NewFakeClock(time.Now()) recorder := recorderWithFakeClock(v1.EventSource{Component: "eventTest"}, caster, clock) var wg sync.WaitGroup wg.Add(100) for i := 0; i < 100; i++ { go func() { defer wg.Done() recorder.Eventf(&v1.ObjectReference{}, v1.EventTypeNormal, "Started", "blah") }() } wg.Wait() caster.Shutdown() } </ins> func TestEventf(t *testing.T) { testPod := &v1.Pod{ ObjectMeta: metav1.ObjectMeta{