Change:
https://prow.ppc64le- Tests in are flaking when run with master golang on ppc64le. After the commit No response The request to APIServer is timing out at The tests are Passing when the timeout value is increased to 200ms The PASS after increasing timeout: Seeing this falkiness only on architecture and when run with golang versions after the commit flakiness /sig testing
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : ### Which jobs are flaking? ### Since when has it been flaking? ### Testgrid link The PASS after increasing timeout: ### Anything else we need to know? Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
i can't seem to find the commit, which PR is that from?
Sorry for the confusion. It is the golang commit
has this an impact on https ? can this be a regression for performance and scalability in Kubernetes? /cc
how close were ppc64le tests to the 100ms timeout before? linux and darwin is nowhere remotely close to the timeout on go1.19 or go master with this diff: there's a ~tiny slowdown of the dial on linux, and a small slowdown on darwin: what does that show before/after on ppc64le?
cc on the question about existing ppc64le timing in
cc - are we running scalability tests on go tip? did we see any increases in CPU or latency or decreases in throughput in the last two weeks?
cc
we run scalability tests on go tip (using fixed version of K8s) but only using x64 architecture for masters and nodes in the cluster. We're also using Kubemark for scale testing go tip in K8s so the results might not be vocal enough about the regression. No visible change in the pod throughput in our tests: ! I looked at some of the most popular <resource, subresource, scope, verbtuples for API call latency we measure in that test but I don't see any visible change at all as well. There might be an increase in apiserver CPU usage judging by the chart below but I'm not sure: ! That said, I'll edit the performance dashboard's config to include more runs to see whether the CPU usage bump is indeed a relatively fresh thing or not.
These failures occur only when run with flag. Without flag, as you said, it is nowhere remotely close to the timeout even on . But with the -race enabled, it is taking ~200ms on after the commit With the just previous commit , it took between 27ms -35ms
Do these tests run with -race flag enabled?
This is an end-to-end scalability test, not a test that one can run through . I also doubt we use the flag when building K8s binaries used in that e2e test given the overhead this flag has on the runtime - see the in article.
The tests this issue mentions are the unit tests run using the , which by default has flag while running a
Even on linux/amd64, I see a steep increase in time after the commit
definitely not, rewrites/instruments code so that it's performance is not at all representative of production use.
if the impact is limited to unit tests with race detection enabled, there's not a notable production impact, but I still asked if the impact was expected or will be optimized in https://go-
looks like there's also a revert CL open at https://go- in case it was needed for performance reasons, so I asked there if this level of impact to race detection code was expected
opened upstream
Meanwhile, Can we bump the timeout value to 200ms ? like we did in
that seems reasonable, since the point of this is not testing performance... though 200ms doesn't seem long enough given the comment in
True :( Will it be too much to ask for 250ms/300ms?
what timeout is used for this same request in other places in this test? it looks like a 1 second timeout is used
True, the same request is using a 1-second timeout at other places in this test!
As the same request is using 1 second timeout at other places, Can we increase it to 1s at
that seems fine to me.
xref
Can you PTAL

Discussion:
client := newClient(true) i := 1 err := wait.PollImmediate(100*time.Millisecond, 5*time.Second, func() (done bool, err error) { <del> result := doer.Do(client, func(httptrace.GotConnInfo) {}, fmt.Sprintf("/echo?message=attempt-%d", i), 100*time.Millisecond) </del> <ins> result := doer.Do(client, func(httptrace.GotConnInfo) {}, fmt.Sprintf("/echo?message=attempt-%d", i), time.Second) </ins> i++ if result.err != nil {