Change:
It is possible to issue a list request which exceeds the duration set by the request-timeout apiserver flag. The request-timeout flag does not appear to be enforced when the apiserver is decoding the response from etcd. Specifically, listing a large number of CRDs from etcd can result in high request latency from using json encoding compared to protobuf encoding. While overall etcd latency is low, the overall listing from storage latency may be over the 1 minute timeout, particularly if the apiserver needs to fetch multiple pages from etcd. We expected the list requests to be subject to the duration specified by the request-timeout flag when this latency is coming from the listing from storage phase of the request. Create 4000 CRs and list all the objects from etcd in 2 pages of 2000 objects. If the time to list 1 page + decode takes around 50s, listing 2 pages will result in the listing from storage phase of the request taking around 1 minute and 40s before the request fails and the client is returned a 500 response code. Logs for apiserver Trace message: Audit log entry: Related issue for supporting protobuf for CRDs: <details</details<detailsAWS </details<details</details<details</details<details</details<details</details
/sig api-machinery CC:
Interesting.. - do you know where exactly we stuck? In the call to storage: we pass the context that has timeout set, so that shouldn't hang. OTOH, for the processing part: (a) I don't know if transformer honors the context - it probably depends on the implementation (b) but the decoding (appendListitem) certainly isn't honoring the context So (b) might definitely be a problem, even independently from from the transformer So I guess what I'm saying is that if listing from etcd finishes fast but the decoding is what takes most of time, then what you described can certainly be a problem. But is that the thing you're observing?
Yes, that is our understanding. Specifically, that the decoding which occurs in appendListItem doesn't respect the context: As a separate follow-up, it would be useful to include decoding time, or at least the overall listing from storage time, as a separate latency annotation, given the audit log entry is only capturing this latency as part of
+1. We need more granular latency tracker especially on the latency spent on decoding. Right now, it only captures etcd round trip and transformer latency.
I would be supportive for both and would be happy to review it if someone opens up a PR for it.
/triage accepted
/assign
/reopen I think we need to reopen it to document how we resolved the unit test failures brought on by adding it
Reopened this issue. <detailsIn response to : /reopen I think we need to reopen it to document how we resolved the unit test failures brought on by adding it Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

Discussion:
ErrCodeResourceVersionConflicts ErrCodeInvalidObj ErrCodeUnreachable <ins> ErrCodeTimeout </ins> ) var errCodeToMessage = map[int]string{

ErrCodeResourceVersionConflicts: "resource version conflicts", ErrCodeInvalidObj:               "invalid object", ErrCodeUnreachable:              "server unreachable", <ins> ErrCodeTimeout:                  "request timeout", </ins> } func NewKeyNotFoundError(key string, rv int64) *StorageError {

} } <ins> func NewTimeoutError(key, msg string) *StorageError { return &StorageError{ Code:               ErrCodeTimeout, Key:                key, AdditionalErrorMsg: msg, } } </ins> func NewInvalidObjError(key, msg string) *StorageError { return &StorageError{ Code:               ErrCodeInvalidObj,

return isErrCode(err, ErrCodeResourceVersionConflicts) } <ins> // IsRequestTimeout returns true if and only if err indicates that the request has timed out. func IsRequestTimeout(err error) bool { return isErrCode(err, ErrCodeTimeout) } </ins> // IsInvalidObj returns true if and only if err is invalid error func IsInvalidObj(err error) bool { return isErrCode(err, ErrCodeInvalidObj)

switch { case storage.IsNotFound(err): return errors.NewNotFound(qualifiedResource, "") <del> case storage.IsUnreachable(err): </del> <ins> case storage.IsUnreachable(err), storage.IsRequestTimeout(err): </ins> return errors.NewServerTimeout(qualifiedResource, "list", 2) // TODO: make configurable or handled at a higher level case storage.IsInternalError(err): return errors.NewInternalError(err)

return storage.NewInternalErrorf("unable to transform key %q: %v", kv.Key, err) } <ins> // Check if the request has already timed out before decode object select { case <-ctx.Done(): // parent context is canceled or timed out, no point in continuing return storage.NewTimeoutError(string(kv.Key), "request did not complete within requested timeout") default: } </ins> if err := appendListItem(v, data, uint64(kv.ModRevision), opts.Predicate, s.codec, s.versioner, newItemFunc); err != nil { recordDecodeError(s.groupResourceString, string(kv.Key)) return err