Change:
From the recent runs of different scalability tests, I have a hypothesis that we are not reusing connections after "429" error is returned to the client. Here is description of one tests that I've seen failed: in 2000-node kubemark all watches were dropped at some point (that's a separate issue that is a bit unrelated here). happened next is that basically all watchers decided to do list + watch, and obvious a bunch of lists failed with 429 error (this is still expected). And we started retrying them. And this is still ok. that point, apiserver started consuming ~30 cores (which also isn't completely unexpected). what IS unexpected to me, is that looking into profile ~70% of the CPU (which was ~20 cores) were spend in "doFullHandshake". And this is a bit strange to me. Do we do something specific on 429 errors (e.g. close the connection) so that we won't be able to reuse it in the future? Or do you have any other hypothesis, why we were mostly doing "full handshakes" at that time?
On the server side, we can use on http server to get more info. Can we do the same thing on the client? I don't see anything obvious, but knowing when a connection from a client is closed (vs. terminated by the server, vs gracefully terminated via Connection: Close) might be useful in debugging these cases. The only thing I can think of is that the kubelet has multiple connections to the master, and so instead of reusing its pooled connections it's asking for new connections all at once (and blowing out the pool size in the client). Can we verify that the kubelet TCP connection pool size is greater than the default (which is 2?). Kubelet should definitely have a number of idle connections proportional to number of watches.
There's definitely a branch in the rest client where I think we don't fully read the response body on a 429. Though not explicitly stated in the docs, I think failing to read the entire body of a request also causes the TCP connection to not be reused. This is a code snippet from the HTTP client's redirect code: I can certainly add something like this to . I'm not sure if this is a problem for HTTP2 though. was this running with HTTP2 enabled? I can't remember the status of that.
I'm pretty sure HTTP2 is off still. , krousey wrote:
Found the doc: I will go ahead and patch this.
Did we add a new error case that didn't catch this? Since this is the 3rd time we've fixed an edge case on reading the entire body, we should probably do what we can to make sure others don't show up.
It's the case where it's 429 or = 500 and there's a Retry-After header. I'm going to add it to the same place where we ensure that the body is closed so that it should always catch it.
k, thanks
Thanks for debugging it (and yes, http2 was switched off in my tests). And looking forward to see the fix. Thanks!

Discussion:
} done := func() bool { <del> // ensure the response body is closed before we reconnect, so that we reuse the same // TCP connection defer resp.Body.Close() </del> <ins> // Ensure the response body is fully read and closed // before we reconnect, so that we reuse the same TCP // connection. defer func() { const maxBodySlurpSize = 2 << 10 if resp.ContentLength <= maxBodySlurpSize { io.Copy(ioutil.Discard, &io.LimitedReader{R: resp.Body, N: maxBodySlurpSize}) } resp.Body.Close() }() </ins> retries++ if seconds, wait := checkWait(resp); wait && retries < maxRetries {