Change:
stop() // Don't defer this. See os.Exit documentation. checkNumberOfGoroutines := func() (bool, error) { <del> // Leave some room for goroutines we can not get rid of // like k8s.io/klog/v2.(*loggingT).flushDaemon() // TODO(#108483): Figure out if we can reduce this //   further (ideally down to zero). if dg := runtime.NumGoroutine() - before; dg <= 4 { </del> <ins> // We leave some room for leaked goroutines as there are // still some leaks, mostly: // - leak from lumberjack package we're vendoring // - leak from apiserve healthz // - leak from opencensus library // Once fixed, we should be able to bring it down to zero. if dg := runtime.NumGoroutine() - before; dg <= 3 { </ins> return true, nil } // Allow goroutines to schedule and die off.

Discussion:
The integration framework runs an apiserver and sometimes other components before each test execution, however, the shutdown is not clean and is leaving a lot of goroutines orphan that keep accumulation between different runs. Some orphan goroutines are connections to etcd, and since the etcd process is shared for all test, we don't know if this can cause contention and flakiness on the job. We are logging the number of goroutines leaked per test binary execution, it is important if we can reduce them and more important we don't regress and this number keeps growing. This issue will record the state as today for reference:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig testing
/cc
hmm, not doing better
I'll take the scheduler part for a deep investigate.
For people that want to get access to the stack traces, just apply this patch, so the framework will dump the output of the goroutines after each execution: [x] Apiserver was shutting down the listener before the hooks could finish, hence the controllers couldn't shutdown gracefully since there was no http server [ ] The storage cacher doesn't differentiate between not read and stopping - Missing integration test reproducing the issue, only reproduced with unit tests [ ] admission plugins can't shutdown gracefully - - there is a PR addressing However, most of the contributions to goroutines leaks are grpc connections that are related to the storage cleanup, there is also a TODO in the code and an open issue I've also tried to reduce the number of connections between storage and etcd, right now, each Storage creates an independent conection to etcd, but sharing the etcd client doesn't work
I think I made some progress with it. I was playing with ttlcontroller test (as it's fairly simple one for the playground and I'm familiar with it, but symptoms in many tests are exactly the same) In I actually managed to reduce amount of leaked goroutines from 1023 to 85.

It's not mergable in this state, but it clearly shows that destroy functions for storage are not called in integration tests. I will try to dig deeper into those and clean this up - but I think the above clearly shows where the problem is. - FYI ^^
is out for review and it's making most of the job here There are still leftovers, but those are mostly oneof's here and there - we should address them independently.
And a bit more longerterm cleaner approach is in
With we're down to 14. I admit that this is one of the simplest tests, but apiserver shutdown is pretty clean once that PR merges.
/honk
! <detailsIn response to : /honk Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
a leftover for the storage cleanup in
Re comment above - yeah, I'm aware, we need to clean it up. FTR:
awesome, we are several order of magnitud better
Looking further into it, in different tests I'm seeing a lot of goroutines with the top of the stacktrace like this: should hopefully address those once it merges... ^^
OK - I think I have now the biggest thing contributing factor to the remaining tests, although I don't yet fully understand it and don't know how exactly to fix it. The main problem is basically that TestServer is not finishing its shutdown: So what exactly is happening is: we run TryDown the TestServer there: this underneath is triggering run of PreShutdown hooks: in the meantime ShutdownInitated signal is signaled: as part of running preshutdown hooks, the bootstrap controller is stopping endpoint reconciler: and then as part of doReconcile, the GET requests hangs forever: By adding time.Sleep(5s) (didn't try smaller values TBH) before the ShutdownInitiated is signalled, the TestServer is shutting clearly: Obviously that's not a solution, but I'm wondering why the GET request mentioned above is hanging forever when ShutdownInitiate is signaled and what we can do with this. - thoughts? [The easiest repro is to run: Adding this sleep basically reduces the number of leaked routines from O(1000) to O(20).]

<--- UPDATE ---In fact, setting ShutdownDelayDuration to 5s has the same effect. So there are two main questions: if setting ShutdownDelayDuration is reasonable why the request actually hangs forever instead of finishing with some error... <--- UPDATE ---I think there must have been something different, as I'm no longer able to reproduce the improvement with my current code. Will try to dig deeper. <-- UPDATE ---OK - the last commit in brings it back to fixed.
I thought that the reconciler hangs because of
Not in this place - here the usual GET request hangs - I will keep looking over the next days.
OK - I understand now and the above isn't really true. The request wasn't really hanging forever. What was happening is that it was being rejected by the WaitGroup handler: and retried by the client-go internals... So I guess from that perspective setting non-zero ShutdownDelayDuration actually makes sense.. - thoughts? ^^
mhm, not sure why setting should help. In general, the server shout stop listening and the HTTP std library should instruct the client's stack to tear down the underlying TCP connection. In that case, the client-go should fail on dialing and stop retrying releasing the server. thoughts?
It's that with ShutdownDelay, the "stopCh" is signaling PreShutdown hooks (that still want to do some work, including API calls) and if we immediately start rejecting that, they will not be able to do that work...
Discussed a bit more offline - I'm going to investigate blocking closing the waitGroup filter on preshutdown hooks. Will try to send a PR in upcoming days.
We're getting there - this is where we are now: I have a bit more in my local branch to upstream, but I need to clean them up and may not have time in the next week or so.
where are we standing today?
Getting close:
A bit more coming soon.
This is where we are now: I understand some of them (though fixing is potentially non-trivial) and need to understand the others (in particular serving). And we should prevent regressions by failing if we have more leaking goroutines - will be looking at that over upcoming weeks.

OK - so here is where we are (with the current check set to not leaking more than 4 goroutines): Comparing to the first comment in this issue - that looks like a significant improvement :) Here is the summary of I think the remaining goroutines in the above files: [x] integration/apiserver/tracing: The goroutines are coming from otel exporter (in particular grpc layer). It seems that something is not closed, although required digging deeper into otel exporter. [ ] integration/serving: The goroutines are coming from the failure case: And they basically are coming from the fact that Queue when created is spawning a goroutine: which is only shutdown on Shutdown() call. But Shutdown in all controllers is only called in defer of controllers of the function that is actually not being run in failure cases. The actual queue that is problematic here is coming from here: [x] testing/integration/controlplane/transformation The goroutines are coming from the fact that we're not closing the kms transformer (grpc goroutines) that is being created. We basically create a grpc service and never close that: Fixing would probably require changing transformer interfact to implement Close method or sth like that.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
- can you please help with the apiserver/tracing test mentioned in
looking
Opened to try fixing the goroutine leak.
It seems the fix worked fine - I'm marking the tracing as fixed - thanks I'm also trying to fix couple other smaller things.
I've opened as a final step. With that PR we're in good enough shape that I'm going to close this issue as further improvements would have negligible impact.