Change:
It is possible to issue a list request which exceeds the duration set by the request-timeout apiserver flag. The request-timeout flag does not appear to be enforced when the apiserver is decoding the response from etcd. Specifically, listing a large number of CRDs from etcd can result in high request latency from using json encoding compared to protobuf encoding. While overall etcd latency is low, the overall listing from storage latency may be over the 1 minute timeout, particularly if the apiserver needs to fetch multiple pages from etcd. We expected the list requests to be subject to the duration specified by the request-timeout flag when this latency is coming from the listing from storage phase of the request. Create 4000 CRs and list all the objects from etcd in 2 pages of 2000 objects. If the time to list 1 page + decode takes around 50s, listing 2 pages will result in the listing from storage phase of the request taking around 1 minute and 40s before the request fails and the client is returned a 500 response code. Logs for apiserver Trace message: Audit log entry: Related issue for supporting protobuf for CRDs: <details</details<detailsAWS </details<details</details<details</details<details</details<details</details
/sig api-machinery CC:
Interesting.. - do you know where exactly we stuck? In the call to storage: we pass the context that has timeout set, so that shouldn't hang. OTOH, for the processing part: (a) I don't know if transformer honors the context - it probably depends on the implementation (b) but the decoding (appendListitem) certainly isn't honoring the context So (b) might definitely be a problem, even independently from from the transformer So I guess what I'm saying is that if listing from etcd finishes fast but the decoding is what takes most of time, then what you described can certainly be a problem. But is that the thing you're observing?
Yes, that is our understanding. Specifically, that the decoding which occurs in appendListItem doesn't respect the context: As a separate follow-up, it would be useful to include decoding time, or at least the overall listing from storage time, as a separate latency annotation, given the audit log entry is only capturing this latency as part of
+1. We need more granular latency tracker especially on the latency spent on decoding. Right now, it only captures etcd round trip and transformer latency.
I would be supportive for both and would be happy to review it if someone opens up a PR for it.
/triage accepted
/assign
/reopen I think we need to reopen it to document how we resolved the unit test failures brought on by adding it
Reopened this issue. <detailsIn response to : /reopen I think we need to reopen it to document how we resolved the unit test failures brought on by adding it Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

Discussion:
// The Write method can be invoked multiple times, so we use a // latency tracker that sums up the duration from each call. ResponseWriteTracker DurationTracker <ins> // DecodeTracker is used to track latency incurred inside the function // that takes an object returned from the underlying storage layer // (etcd) and performs decoding of the response object. // When called multiple times, the latency incurred inside to // decode func each time will be summed up. DecodeTracker DurationTracker </ins> } type latencyTrackersKeyType int

TransformTracker:         newSumLatencyTracker(c), SerializationTracker:     newSumLatencyTracker(c), ResponseWriteTracker:     newSumLatencyTracker(c), <ins> DecodeTracker:            newSumLatencyTracker(c), </ins> }) }

} } <ins> // TrackDecodeLatency is used to track latency incurred inside the function // that takes an object returned from the underlying storage layer // (etcd) and performs decoding of the response object. // When called multiple times, the latency incurred inside to // decode func each time will be summed up. func TrackDecodeLatency(ctx context.Context, d time.Duration) { if tracker, ok := LatencyTrackersFrom(ctx); ok { tracker.DecodeTracker.TrackDuration(d) } } </ins> // AuditAnnotationsFromLatencyTrackers will inspect each latency tracker // associated with the request context and return a set of audit // annotations that can be added to the API audit entry.

responseWriteLatencyKey     = "apiserver.latency.k8s.io/response-write" mutatingWebhookLatencyKey   = "apiserver.latency.k8s.io/mutating-webhook" validatingWebhookLatencyKey = "apiserver.latency.k8s.io/validating-webhook" <ins> decodeLatencyKey            = "apiserver.latency.k8s.io/decode-response-object" </ins> ) tracker, ok := LatencyTrackersFrom(ctx)

if latency := tracker.ValidatingWebhookTracker.GetLatency(); latency != 0 { annotations[validatingWebhookLatencyKey] = latency.String() } <ins> if latency := tracker.DecodeTracker.GetLatency(); latency != 0 { annotations[decodeLatencyKey] = latency.String() } </ins> return annotations }

"k8s.io/apimachinery/pkg/runtime/schema" "k8s.io/apimachinery/pkg/watch" "k8s.io/apiserver/pkg/audit" <ins> endpointsrequest "k8s.io/apiserver/pkg/endpoints/request" </ins> "k8s.io/apiserver/pkg/storage" "k8s.io/apiserver/pkg/storage/etcd3/metrics" "k8s.io/apiserver/pkg/storage/value"

default: } <del> if err := appendListItem(v, data, uint64(kv.ModRevision), opts.Predicate, s.codec, s.versioner, newItemFunc); err != nil { </del> <ins> obj, err := decodeListItem(ctx, data, uint64(kv.ModRevision), s.codec, s.versioner, newItemFunc) if err != nil { </ins> recordDecodeError(s.groupResourceString, string(kv.Key)) return err } <ins> // being unable to set the version does not prevent the object from being extracted if matched, err := opts.Predicate.Matches(obj); err == nil && matched { v.Set(reflect.Append(v, reflect.ValueOf(obj).Elem())) } </ins> numEvald++ // free kv early. Long lists can take O(seconds) to decode.

return nil } <del> // appendListItem decodes and appends the object (if it passes filter) to v, which must be a slice. func appendListItem(v reflect.Value, data []byte, rev uint64, pred storage.SelectionPredicate, codec runtime.Codec, versioner storage.Versioner, newItemFunc func() runtime.Object) error { </del> <ins> // decodeListItem decodes bytes value in array into object. func decodeListItem(ctx context.Context, data []byte, rev uint64, codec runtime.Codec, versioner storage.Versioner, newItemFunc func() runtime.Object) (runtime.Object, error) { startedAt := time.Now() defer func() { endpointsrequest.TrackDecodeLatency(ctx, time.Since(startedAt)) }() </ins> obj, _, err := codec.Decode(data, nil, newItemFunc()) if err != nil { <del> return err </del> <ins> return nil, err </ins> } <del> // being unable to set the version does not prevent the object from being extracted </del> if err := versioner.UpdateObject(obj, rev); err != nil { klog.Errorf("failed to update object version: %v", err) } <del> if matched, err := pred.Matches(obj); err == nil && matched { v.Set(reflect.Append(v, reflect.ValueOf(obj).Elem())) } return nil </del> <ins> return obj, nil </ins> } // recordDecodeError record decode error split by object type.