Change:
cc
This is still hapening. Assigning Isaac for triage.
Three times in the last three days, BeforeSuite failed (filed ). Otherwise, just Namespaces [Serial] should delete fast enough (90 percent of 100 namespaces in 150 seconds) is just totally borked (filed ). what prevented you from triaging this yourself? It'd be good to know so I'm less in the critical path.
Sorry, I read this wrong. I was looking at the gke-serial suite. Triaging now.
It looks like massive failures are correlated extremely strongly with failure of Etcd failure [Disruptive] should recover from SIGKILL Every time that test passes, everything else (except ) passes; every time it fails, the whole suite goes south. Reassigning to for triage.
sorry for delegating the work to you, but I completely didn't have time to do it and nobody else hasn't volunteered to do it for a while. it seems you are the author of the test, so assigning you.
No problem Just wanted to make sure there wasn't some information missing.
I'm looking at this today
It looks like the token controller is not recovering from etcd failing. The test namespace is also stuck in a terminating phase. I'll continue to investigate
Actually lot's of watch related things are broken. I can create/delete objects in the api but watch is broken. Trying to establish a watch in this state causes: Restarting the apiserver fixes the problem. I can start a watch again: cc
Not particularly useful apiserver logs Also none of the components are trying to reestablish watches so I'm assuming that they still have watches open but are not receiving anything.
startCaching is trying to lock here, but something is holding the lock. I'm looking through a stack trace right now for the offender. This is exactly where I'd expect a problem to be from the description of "bringing etcd down and up repeatedly".
Ah I think what happened is, when etcd stays down, a List never successfully finishes, and therefore it never calls the OnReplace function, and therefore the lock is never unlocked, and therefore startCaching can't exit.
There's lots in apiserver log to support this: Additionally I manually looked through the stack trace and confirmed that all of the stuck calls to Watch are stuck waiting to RLock and not some point afterwards.
This is pretty bad, etcd going down for just a bit could freeze up apiserver until it is rebooted.
I think I can make a fix.
Thanks for looking into this.
Yeah, I repro'd in a UT. Fix coming shortly.
I assigned to but if anyone wants it merged sooner than monday, feel free to review.
Many thanks to for handing me a broken server to look at, it made this super easy to find.

Discussion:
stopCh: make(chan struct{}), stopWg: sync.WaitGroup{}, } <ins> // See startCaching method for explanation and where this is unlocked. </ins> cacher.usable.Lock() <del> // See startCaching method for why explanation on it. watchCache.SetOnReplace(func() { cacher.usable.Unlock() }) </del> watchCache.SetOnEvent(cacher.processEvent) stopCh := cacher.stopCh

} func (c *Cacher) startCaching(stopChannel <-chan struct{}) { <del> // Whenever we enter startCaching method, usable mutex is held. // We explicitly do NOT Unlock it in this method, because we do // not want to allow any Watch/List methods not explicitly redirected // to the underlying storage when the cache is being initialized. // Once the underlying cache is propagated, onReplace handler will // be called, which will do the usable.Unlock() as configured in // NewCacher(). // Note: the same behavior is also triggered every time we fall out of // backend storage watch event window. defer c.usable.Lock() </del> <ins> // The 'usable' lock is always 'RLock'able when it is safe to use the cache. // It is safe to use the cache after a successful list until a disconnection. // We start with usable (write) locked. The below OnReplace function will // unlock it after a successful list. The below defer will then re-lock // it when this function exits (always due to disconnection), only if // we actually got a successful list. This cycle will repeat as needed. successfulList := false c.watchCache.SetOnReplace(func() { successfulList = true c.usable.Unlock() }) defer func() { if successfulList { c.usable.Lock() } }() </ins> c.terminateAllWatchers() // Note that since onReplace may be not called due to errors, we explicitly

package storage_test import ( <ins> "fmt" </ins> "reflect" "strconv" "testing"

} } <ins> type injectListError struct { errors int storage.Interface } func (self *injectListError) List(ctx context.Context, key string, resourceVersion string, filter storage.FilterFunc, listObj runtime.Object) error { if self.errors > 0 { self.errors-- return fmt.Errorf("injected error") } return self.Interface.List(ctx, key, resourceVersion, filter, listObj) } </ins> func TestWatch(t *testing.T) { server, etcdStorage := newEtcdTestStorage(t, testapi.Default.Codec(), etcdtest.PathPrefix()) <ins> // Inject one list error to make sure we test the relist case. etcdStorage = &injectListError{errors: 1, Interface: etcdStorage} </ins> defer server.Terminate(t) cacher := newTestCacher(etcdStorage) defer cacher.Stop()