Change:
REGISTRY ?= gcr.io/kubernetes-e2e-test-images GOARM=7 QEMUVERSION=v2.9.1 <del> GOLANG_VERSION=1.12.4 </del> <ins> GOLANG_VERSION=1.12.5 </ins> export ifndef WHAT

Discussion:
Something has changed between 320 and 321 runs and the request latency has increased significantly. The graphs below shows request latency (non watch) over time for a single test run: : ! : ! In my opinion they looks significantly different: in 320 most of the spikes are lower than 0.8s while 321 most of the spikes exceeds 1.6s. I checked few runs below 320 and have latency pattern similar to 320 and few runs after 321 and they have similar pattern to the 321. We are seeing similar latency growth in kubemark tests. The most significant change between 321 and 320 was go 1.11.5 -1.12 migration. /cc
/sig scalability
With we are trying to prepare small scale repro that we can use for further debugging.
was also in that range (cc for verification that nothing in that PR increased server load in non-alpha clusters)
did tests at head with reverting to go1.11 and metrics seemed better He is working on confirming that and should paste results then, but it seems to be pretty clear that it's go regression. We're also working on providing smaller-scale repro case to work closer with golang team (they used to be really helpful when debugging regressions in previous versions (IIRC 1.9 is where there were previous one).
I double checked and didn't find anything suspicious. The alpha function wasn't invoked in that window, and the PR is a no-op to non-alpha clusters
Regarding what wrote: ! I was mostly investigating the regression on kubemark-5000. Some of its latencies increased x3. I run two tests: one with checkout, and the other one with some commit checkout but without (Update to use golang 1.12). The 99th latency percentile of node listing was 3400.152ms in first case and f 952.576ms in second one.
Look into other metrics (I can't remember which exactly, delete pods? post pods? or sth like that) - which increased by almost 10x.
This is the graph that I had on my mind, but yeah, it's more like 4x not 10x. But still quite huge drop: !

Reproducing the issue: kubernetes repo - First broken version (go1.12) Last working version (go1.11) perf-tests repo - This repo is home for clusterloader2 - tool used for running the scalability tests. kubernetes directory run “make quick-release”. This will build kubernetes based on checkout version. GCE cluster. kubemark test. This step creates kubemark cluster (on top of the cluster created in step 4), run cluster loader test and tear down the cluster. The test run metrics will be exported to directory provided by the --report-dir flag. Inside this directory there will be APIResponsivenessload file containing 50, 90, and 90 percentiles of all apiserver call latencies. The latencies affected the most are: Listing nodes { "Resource": "nodes", "Scope": "cluster", "Subresource": "", "Verb": "LIST" } Pod deletion { "Resource": "pods", "Scope": "namespace", "Subresource": "", "Verb": "DELETE" } Listing services { "Resource": "services", "Scope": "cluster", "Subresource": "", "Verb": "LIST" } I would suggest their 99th percentiles as the issue metric. As for comparison: -Go1.11 run: Listing nodes - 162.409ms Pod deletion - 41.302ms Listing services - 28.277ms -Go1.12 run: Listing nodes - 293.724ms Pod deletion - 56.2ms Listing services - 46.183ms down GCE cluster
We're trying to dig deeper into it to both root cause it to a smaller part of the system, but it is quite tricky. Unfortunately, we don't have much more than guesses right now, though. Some observations: I looked into kubemark-5000 runs into DELETE pods calls, and it seems that all of them for which we have traces (500ms+) - there are ~5000 of those in each of recent runs [previously 99th percentile was <200ms), there is a significant big of "Object deleted from database". That may suggest some etcd issues, but we don't see logs related to long-running-calls on etcd side.

That may suggest some issues on the etcd client side, but we don't have anything concrete yet.It's also possible that golang upgrade changes something in serialization/deserialization - that's a second hypothesis (we've seen a bunch of traces of "Decoded delete options" (deserialization is also done in etcd client)), but I'm not convinced this is the case as of now
We've spent significant time with golang folks and already have some findings. To summarize: the microbenchmark from enabled Go team to find some regression on their side - the faulty commit from golang is: to cite Go folks: "To summarize the change: I removed the "fast" path for small page-level allocations (which currently always need to acquire the heap lock) in order to simplify code for subsequent changes. In effect, span allocations now all go through a balanced tree rather than just a linked list for small spans (I think it was all spans under 128 pages?)." We aren't fully convinced that this is for sure the same thing that is causing regression in our scalability tests, but analyzing traces, profiles, golang tracesetc. suggests that problems are indeed around memory allocations or GC. So with high probabilty this is the same thing. We are discussing with the way to prove it's the first thing and working with the to see what we can do with it.
It appeared that the commit isn't the most faulty one for us - running scalability tests against that commit doesn't show significant drop in metrics. In the lack of better ideas, we decided to bisect changes in golang by running full scalability suite on 5k-node cluster against them. This will take couple more days though...
Just to update this thread - we're bisecting Go 1.12 branch and there aren't that many remaining commits. We already know that: is a good one is a bad one I hope that we will know the exact faulty commit on Monday.
Looking through the following stand out for me /me places bets on one of those 2 :)
That is our bet too - we're working with Go folks to confirm and fix/mitigate this.
Any luck? (curious!)

I'm sorry for delay with update here - the current update is as following: after the next round of bisections, we know that: is a good one is a bad one During my discussions with golang folks, the outcome was that we believe that is the faulty one, as it adds a bunch of syscall to second-slowest allocation path (built on top of the previous one) should only help So the next step was to verify and indeed it is a bad one (and actually as expected, the metrics were worse than in the so far confirmed oldest bad commit: ) This is what we already know. What we're doing now is that golang folks provided us two patches that they think should mitigate the issue and I'm trying to confirm if that is really true. We should have results of this one later today. The two potential outcomes are: it solves our issue; then I will be working with golang folks to patch 1.12 with those it doesn't solve our issue: then I will be working on final confirmation that it's exactly this commit (by checking the previous one, and the next one to confirm it really helps) and working further with golang folks on what else we can do to mitigate it (and also I will try to understand deeper what exactly is happening there, I just have some rough understanding now).
Very very cool! thanks
cc
Just to give an update on that.

we fully confirmed that the regression is due to (using go1.12.1 with manually reverted that commit reverts the behavior) however, according to golang fokls that commit was fixing some things, and there are couple commits built on top of them after 1.12, so it's not-trivial to revert it (we are still looking for other options)the patches that golang team provided helps, but don't fix the problem (they roughly reduce the problem to 50%) yesterday I got another two patches that may help and we're validating them nowin the meantime we're also working with golang folks to try gathering more useful data for them for debugging - they already have some patches to apply in golang and I'm working on piping them through k8s and our build scripts - I hope I will have them ready today for an over-night run
To give an update on that - the current state of it is as following: that particular commit can't be just reverted, as it was fixing some issue however, golang folks have a PR that is mitigating that issue in a different way we have verified with custom build of that fully mitigates regression on our side The current plan is as following: has been merged to head of Go [that is the mitigation commit] it will be cherrypicked to 1.12 branch additionally in 1.12 we will revert the offending commit: We expect a new patch release of Go with those two commits to be released on May 1st. is currently my main point-of-contact coordinating that on golang side.
thanks for the update ! we will still be in 1.15 alpha* phases so that's great news.
- the trick is that go 1.12 is also in 1.14, so we should also cherrypick 1.14 branch - currently there is visible regression in 1.14
agree. i'd support that
The commits that we were waiting for are already there: Now we are waiting for next 1.12 patch release of Go to update Kubernetes to use, the ETA is May 1st. - given that I will be on vacation for the rest of this week, would you be able to help with that?
yes i can, please let me know what needs to be done (so i can figure out if i have access before you head out)
- thanks a lot.

I didn't do upgrade of go for 2 years or so, so I'm not entirely sure how (if?) the process changed in the meantime. - was doing the last one: , so I'm sure he will be able to help with that.
Ah just update to latest golang when it gets out. yes, i can do that. i was wondering if you wanted me to kick off some CI runs or something. Have a good vacation
- thanks a lot! Yes - it wasn't clear, but I was testing that and it helps us, so no need to run another tests (we will get final confirmation from postsubmits).
I can also help out to get this upgrade in when it’s released. Thanks for all your work on this
Looks like 1.12.5 went out this afternoon. PR open (), but expect tests to fail: Waiting for the docker hub image to publish. The PR is merged, so this should happen shortly. Need a new version of the bazel rules_go to be published.
Just to clarify the above (from : "oh, so I saw the auto update ran here: but it looks like the update for the tags hasn't merged yet: "
Yea, it takes some time for the images to get published.
Looks like new images are live.
Reopening this so we can verify that everything is back to normal in CI
/assign
For me it's pretty clear that this was resolved: http://perf- http://perf- Now there is one more thing we should do, which is cherrypick it to 1.14 branch. ^^
Opened pick:
has SIG Scalability done any one-off custom scale/perf testing with the new golang patch release and the k8s 1.14 branch content? Do you see this cherry pick as a 1.14.2 release blocker? Or would a week or two of focused testing with updated golang in the 1.14 k8s branch and a 1.14.3 release by end of May be a sufficiently safe yet expedient response?
We were doing a bunch of tests to discover the real regression in golang 1.12 - the regression happened within 1.14 release timeframe and the bisection was done on that branch. So yes - I'm 100% confident that it helps.

Tough question - I would love to see it in 1.14.2, otoh given how long it took to debug and fix it, I can see release team pushing it to 1.14.3. But if we could have that in 1.14.2, it would be extremely helpful.
just merged and we let 1.14 soak for one or two days before we cut . Let's keep this issue open until we have some data on the latencies after that merge. SIG Scalability, can you please provide some data / graphs about the latency in a (couple of) days for the branch?
is there any evidence we'll be able to observe in for validation? Or is it possible for SIG Scalability to trigger any custom runs now that the cherry-pick has merged to the 1.14 branch?
We aren't running large tests on release branches due to cost of them... We don't have any free quota to run it now, due to huge number of regressions that we have at head that we're currently debugging. But I'm 100% confident with this change. What I was personally doing in the past: I was running tests against commit when we bumped to 1.12.0 with 1.12.5 (which was from beginning of March, so still in 1.14 timeline) and the effect was exactly the same as the effect against head we got after merging it. I also understand changes in Go between 1.12.4 and 1.12.5 and their effect on performance And we also understand this regression, and why the fixes help, which has nothing to do with any changes that are happening to kubernetes. So I'm entirely sure that if you didn't observe any correctness issues, from performance perspective it is good-to-go.
Thanks for your answer!
With this being fixed in head and 1.14.2 released with the fix, I'm closing this one as fixed.