Change:
various Oct 30 2023 6pm pst No response Looks like different test are failing with connection timeout errors: One of the tests triggered a core dump: No response /sig
are both suffers from pkg/registry and is from compoent-base, which likely same reason from `
/retitle failing-test: pkg/kubelet/eviction TestManagerWithLocalStorageCapacityIsolationOpen
/sig node
TestManagerWithLocalStorageCapacityIsolationOpen only failed in one of the PRs, so I don't think that specific test is the problem.
ok, rename as see fit. just put owning sigs please for triage.
TestStoreDeleteCollectionWithContextCancellation is the one that seems to be failing across all the failing PRs, so let's start there first. /retitle failing-test: TestStoreDeleteCollectionWithContextCancellation /sig api-machinery
triage link at shows this just starting (happening in CI jobs as well)
image used in the unit test job didn't change around the timeframe this was seen ()
looking for jumps in package unit test runtimes when this failure started happening to see if something is taking longer or stressing the system: ! - cc ! - cc ! - cc Could any of those have introduced changes that could be stressing or exhausting listener connections in unit tests?
the PR in question introduced a poll function around an API call that is unit tested with a fake client i.e. the poll is unit tested, but the fake client has reactors that return error or success right away, so in theory the poll should not be stretched to 1 retries with 1 connection attempts per fake client. while testing locally i did not notice any significant time difference after or the prior i can do some more checks to see what's going on.
merged right at the time the flakes were seen DeleteCollection invokes List, which was modified in that PR to cancel more aggressively cc
I can reproduce about a 10-20% flake rate at HEAD with: failures look like: or while testing that package with , I see ~1300-1500 open connections with this:
Let's revert it then. I looked into it and I don't know from a quick glance why it's failing and I don't have time to debug it now. I opened
Okay, it looks like we need more debugging to open this PR
one optimization to strip some ~40 fake client connections and 2 seconds from a test:
sounds good, pointed at that commit as well
/triage accepted

Discussion:
ErrCodeResourceVersionConflicts ErrCodeInvalidObj ErrCodeUnreachable <del> ErrCodeTimeout </del> ) var errCodeToMessage = map[int]string{

ErrCodeResourceVersionConflicts: "resource version conflicts", ErrCodeInvalidObj:               "invalid object", ErrCodeUnreachable:              "server unreachable", <del> ErrCodeTimeout:                  "request timeout", </del> } func NewKeyNotFoundError(key string, rv int64) *StorageError {

} } <del> func NewTimeoutError(key, msg string) *StorageError { return &StorageError{ Code:               ErrCodeTimeout, Key:                key, AdditionalErrorMsg: msg, } } </del> func NewInvalidObjError(key, msg string) *StorageError { return &StorageError{ Code:               ErrCodeInvalidObj,

return isErrCode(err, ErrCodeResourceVersionConflicts) } <del> // IsRequestTimeout returns true if and only if err indicates that the request has timed out. func IsRequestTimeout(err error) bool { return isErrCode(err, ErrCodeTimeout) } </del> // IsInvalidObj returns true if and only if err is invalid error func IsInvalidObj(err error) bool { return isErrCode(err, ErrCodeInvalidObj)

switch { case storage.IsNotFound(err): return errors.NewNotFound(qualifiedResource, "") <del> case storage.IsUnreachable(err), storage.IsRequestTimeout(err): </del> <ins> case storage.IsUnreachable(err): </ins> return errors.NewServerTimeout(qualifiedResource, "list", 2) // TODO: make configurable or handled at a higher level case storage.IsInternalError(err): return errors.NewInternalError(err)

return storage.NewInternalErrorf("unable to transform key %q: %v", kv.Key, err) } <del> // Check if the request has already timed out before decode object select { case <-ctx.Done(): // parent context is canceled or timed out, no point in continuing return storage.NewTimeoutError(string(kv.Key), "request did not complete within requested timeout") default: } </del> if err := appendListItem(v, data, uint64(kv.ModRevision), opts.Predicate, s.codec, s.versioner, newItemFunc); err != nil { recordDecodeError(s.groupResourceString, string(kv.Key)) return err