Code change:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Cluster was upgraded from 1.19.5 to 1.20.1 using kubeadm. After that, kube-apiserver cpu usage went from 50% to 300%. What you expected to happen: Expected that apiserver has the same cpu usage. How to reproduce it (as minimally and precisely as possible): Update a cluster from 1.19.5 to 1.20.1. Anything else we need to know?: The cluster is running on sbc hardware, thats why the original 50% cpu usage, and thats why 300% cpu usage is noticeable and causes service disruption. Environment: Kubernetes version (use ): (the apiserver is rolled back to 1.19.5)Cloud provider or hardware configuration: bare metal on SBC hardware. OS (e.g: ):Kernel (e.g. ):Install tools: kubeadm Network plugin and version (if this is a network-related bug): kube-router Others: Please check profiling of each apiserver.
/sig api-machinery
please do a and show the contents here. (hide sensitive IPs if needed). this is the static pod for the kube-apiserver. also, anything interesting in the logs of the api server?
On another cluster I could not reproduce the issue. But, on the original one, it happens again if i switch to the newer apiserver image. So, now I am bisecting apiserver to find the commit since it behaves this way, hopefully that will help. I could not spot any interesting in the logs. I will be able to provide more after the bisecting procedure.
post the above output whenever possible.
The static pod manifest has been moved to avoid kubelet starting it: Still bisecting, trying to find the cause. May it relate to the go1.15 upgrade? Building the apiserver with go1.15.6 seems to be working, while building with 1.15.5 results in a wrong binary. But, still I am working to confirm this.
With apiserver built with go1.15.5 I get 300% cpu usage most of the time, and the following log entries appear:
unclear whether we have perf tests to catch similar problems: /sig scalability for other readers, go1.15.5 is the version used to build 1.20: /sig auth this and similar come when doing rest storage on various apis. e.g.
/retitle performance regression: 1.20 apiserver has high cpu usage
/kind regression
Definitely it seems that the binary built with go1.15.5 is broken. It ends up on 300% cpu usage and with log messages posted above. Note that I am building apiserver this way: Building with go1.15.6 is better, even if not all of the time. Perhaps I am on the wrong way?
this is a detail i've missed from the OP. since we don't have k8s ARM tests, it is likely why we are not catching this and it could be a golang compiler issue on ARM only. in the past we have seen some weird issues from ARM users, that have hinted about compiler bugs.
Sorry for that, I did not mention that explicitly. If it turns out to be a compiler bug, will it trigger a golang upgrade for k8s images?
I will make a build with golang 1.15.2 too and post the results later.
if we know that a certain golang version is buggy, we can update a specific k8s version to use a golang version that works. in this case the latest 1.20.x k8s can be updated to not use 1.15.5.
Simply compiling 1.20.1 with golang 1.15.2 still produces misbehaving binaries. So, I will do a bisecting again with building binaries as documented, and try to find the bad commit this way. This will take longer on my hardware.
So, apiserver behavior is a bit random regarding this issue. What is true that on my hardware (orange pi pc) the apiserver needs around 85 cpu seconds to boot up, after that it consumes ~10% cpu time. When a build is bad, apiserver passes the 85 cpu seconds and still uses more cpu than 10%, sometimes even 300%. So, based on this, bisecting shows is first bad commit.
Reverting that commit on 1.20.1 makes apiserver behaving as before. Consistently it finishes its bootup in 90 seconds, and then cpu usage gets to its normal 10-50%, it depends on load-balancer.
By the way, the problem is not just high cpu usage, but during bootup and high cpu usage, apiserver stays in not ready state, thus kubelet restarts it after failed liveness probes. So it may cause service outage.
we had long discussions in the past where RPI users complained about the kubeadm probe defaults. we claimed something else is not right if the probes are failing, so i think you are confirming that. i can ping the release team and api-machinery about this ticket, but note post the holidays responses will be slow.
/retitle performance regression on ARM: 1.20 apiserver has high cpu usage
I suspect, that the mentioned commit fixes things in a way that it actually places more cpu load, I mean expression evaluates to true more often. The attached profiling data may be a proof for this.
It seems that on slower hardware, operations are distributed into more sample periods (time.Millisecond), which makes more calls to the expensive metrics.Observe functions. Before the mentioned fix, due to quantization bug, those calls were not made every time.Milliseconds. Now they are, and profiling data proves this. Can we raise the samplePeriod, can it be made configurable?
sig api machinery can provide feedback on this.
is that accurate? my reading of the fix is that the quantize method returns results with the same granularity. The bug that was fixed was that quantization could return numbers that resulted in negative calculations if wall clock time changed, and now it respects the monotonic clock.
Perhaps not. That was just my assumption. But still, profiling data shows (at least for me) that metric.Observe calls take most of the time, while reverting the mentioned commit solves my problem. Somehow less calls are made to them.
/assign
We have numerous K3s users reporting perf issues (and crashes in the cronjob controller) on armv7l. I can reproduce it locally on my Pi nodes as well. I hadn't gotten around to digging into it yet, so I'm glad someone else has done so. We also had issues getting arm to build properly starting with Kubernetes 1.19.4; some builds would crash with what looked like a corrupted stack on armv7l unless we set for the build. See: Prior to doing that it was fairly random if a build would work or not; changing unrelated bits of code (ie bumping a vendored module version) would give us a clean build, and then changing something else would break it, and then changing a 3rd thing would give us a clean build again. It felt very much like golang arm toolchain issues. This sounds similar to what reported with good vs bad builds.
I've a little debugging: And I got strange results, a snippet from stdout: Check out the high on the 4th line. That causes a long-long loop to start. And there are more of such lines. If I refactor the code to remove the inline func() and do the Unlock() and the end of the block, this behavior goes away, apiserver goes back to normal. May it be a compiler bug?
earlier you've mentioned: Building the apiserver with go1.15.6 seems to be working, while building with 1.15.5 results in a wrong binary But, still I am working to confirm this. did you confirm that the newer toolchain does not produce code that exhibits the problem? seems likely. in the past i've seen at least one compiler bug-like report from a k8s user (on ARM)
Unfortunately, I think I got confused regarding compilers. Now it seems that I got the same results with go1.15.2, 1.15.5, 1.15.6. So, now I think it does not matter which toolchain I use.
I've applied the following patch on release-1.20: And this produces strange results: It can be observed that from the third line, the function receives strange values. Later normal values appear, but sometimes the calculated is a large positive integer, which triggers a long running loop. Unfortunately, I am not familiar with golang's internals, and could not reproduce this in a few lines of code. But indeed, it seems that something weird is happening. If I place a before the call to , then the function gets the correct value passed all the time. How to proceed?
I think this is an arm32 int64 division overflow/underflow issue? I'm currently testing a minor change to confirm/deny this hunch, but building for arm takes forever so it's still running.
It looks like the time value is being corrupted when being passed into quantize somehow, would be nice to try and reproduce with a very small executable. k3s has a long history of arm32 issues like this that I suspect is a core golang issue, but hard to reproduce and isolate, arbitrary code changes could change the result.
you could try running something like this, which exercises the local logic of the old and new quantize methods and checks that they output roughly similar values (some drift is expected because one uses wall clock and the other uses monotonic clock) I was able to run this for several minutes on linux/amd64 with no problematic values output
Yes, it looks like parameter corruption rather than a division problem. very strange.
Yeah my simple change to the quantize didn't make any difference. That combined with the strange output from the debug printing does indicate some sort of corruption to the when time struct that's being passed in, rather than bad math on the parameter.
Presumably all the unit tests are passing on the arch & OS you are using.
But there are no unit tests focused on that. It's on the to-do list.
If we created unit tests for this file, would somebody be running them on the arch & OS involved here?
I put in a check in quantize to grab a stack trace if when is before t0:
This seems like papering over a compiler problem, but this fixes it for me:
: that is interesting info. I would prefer to not merge that change, we have another mystery where the monotonic time seems to go backwards, and I would hate to lose half the debug info from that. I will add unit tests for the sample-and-watermark histograms; can somebody please run them on the arch & OS involved here?
The time going backwards stuff all seems to be related to VM time sync, isn't it? In this case the compiler seems to be doing something that's corrupting the time struct out from under the anonymous function; the fact that it appears to be going backwards is not (as far as I can tell) due to host time actually jumping backwards into the dark ages with a weird TZ offset. FWIW I considered moving the Before check-and-err into the quantize function, but it wasn't necessary for a minimal fix for this issue don't I left it alone. I have a couple armhf (Pi3b/Pi4b) systems I can run tests on, assuming it's not too difficult to set up.
I am not sure whether the monotonic time going backwards is the same bug. It was observed on Intel arch. I suspect another way to paper over the compiler bug would be to switch the argument to from pass-by-value to pass-by-reference. I suspect a compiler bug would be a valuable thing to find. I am working on unit tests for this package. They will involve dependency on the Prometheus go client library, but nothing more. Will that be simple enough?
The incidents I have personally seen or helped users with time sync issues have all been VMs on Intel. If there are non-virtualized platforms with this issue it'd be interesting to see. Simply preventing the sampling time bucket from going backwards seems reasonable, which appears to be what the fix was attempting to do. Changing it to a pointer is an interesting idea. I can give it a try later this evening. I was figuring I'd have to run the entire Kubernetes test suite, so anything less than that is great!
Your small program is running on arm for minutes now without any errors. It needs more to trigger the compiler bug.
I will be able, though I will need instructions how to do that.
Here is a typescript of me running the new unit test: (the test does exercise the case of the clock going backwards a little) That's using the branch in
On ARM, I get similar results:
/remove-sig auth
: dang. Can you try it a few times? If that does not provoke the problem, perhaps you can edit the test to do a lot more iterations? You can also cut down on the warning noise by making a much smaller fraction of the cases have time going backwards.
Running multiple times results in some failing. Sometimes there are cases when 3 consecutive runs are successful. So basically, the test triggers the issue, but not deterministically. Even with iterations set to 1000, there are still successful test runs.
Is that close enough to submit as a compiler bug report? BTW, what do the failures look like?
is the logic non-flaky on AMD64? for a compiler bug, the maintainers are likely to request the failing code disassembled, so that they don't have to clone k8s, or a minimal code example that fails.
I am getting non-flaky results on this: I think I have gotten the test to be about as simple as makes sense for a kube unit test. I will have to leave the compiler bug hunting to people more familiar with ARM.
I tried to make a stdlib-only file that exercises the same call patterns w.r.t. variable scope as the original at what does that do when run on ARM?
Interesting, running the tests by on an rpi4 (using arm32) I am able to trigger two different issues: clock is NOT monotonic and can go backwards time IS corrupted periodically when being passed into I am also not able to trigger any issues running those tests on Darwin (using x86_64). However, running the stand-alone go file by I am not able to trigger any problems on the rpi4
: sorry if it wasn't clear, but the test introduced by uses a fake clock that is set backwards in some cases just to test how the metrics code reacts to that.
Same here, on an OrangePI armv7 board, the single stdlib-only test does not produce any errors.
Later I will do a test on arm64 as well.
woops, ah thank you I had to add to use the flag with go test to see those messages in Darwin and missed that part of the test code. At least the arm32 corruption issue is reproducible with that unit test, but I suspect that attempts to reduce the code space for the problem may cause it to go away, so maybe an issue of memory management or executable code layout. Unfortunately seems like debugging tools can be hard to come by on arm32 (eg no delve), am open to any ideas or suggestions on how to dig into it more
BTW, I updated to always log an info message explaining what's up with the warnings.
i don't know how to handle go mod imports with and some of the golang intrinsics are weird, but you could instead: build the unit test as a binary with the binary and for the object code of interest log a ticket @ golang and share the dissasembly you might also be able to use gdb over the test binary - set some breakpoints and try to catch what instruction is stomping the register / stack address that holds the value that ends up being bogus:
Thanks a bunch Was just looking into the compiled test, that is a great guide & will submit an issue with golang.
On arm64 the tests succeeded multiple times, no failures.
Compiled the test code for i386 resulted in: On i386 the test code triggers the bug too.
Different GOGC settings results in different test failure rate:
I am able to reproduce that. ... and if I build without the variant GOARCH then the test passes all 300 times.
When I merge and , the problem is visibly gone on the 386 arch on linux.
should we merge the pass-by-reference bit of in the meantime until golang/go is resolved?
I would say that we could wait a few days for response on the bug report. On the other side, I dont know how fast a resolution will get into a stable golang release, and what is needed for k8s to be built using that. Can these take a long time?
yes, I would not expect it faster than O(weeks), potentially O(months) if the golang investigation takes a while
It would be great if the fix could be merged and backported in time for 1.20.2; as it currently stands 1.20 is basically unusable on arm.
BTW, contains only the pass-by-reference change and logging of an error if there is a wild jump anyway; I would recommend taking all of that PR.
the deadline for cherry-picks to be merged for 1.20.2 is tomorrow
i wonder if it's also related w/ , the latter issue seems to be stambling on the memory pressure tho. but cpu&memory consumption can be also connected.
I think yes, I've seen similar log messages.
For what it is worth this seems like a very high impact bug, as it can potentially effect 64-bit architectures also. The following example of the bug that was submitted as part of the fix also fails on Darwin/x86_64: It seems like the best mitigation until a new version of golang is released is to disable optimizations for the compiled binaries.
I created as a cut-down alternative to , containing only the work-around and not the checking and logging for the argument passing problem.
: What do you recommend for how to proceed on this? We should probably get something into 1.20.2
agree. I'm fine with a temporary commented pass-by-reference workaround.
/triage accepted
ping , On arm64 I got the same error like . and kube-api-server restart many times.
the error you are getting looks like a connection problem, so not clear if related. the bug discussed here is in golang. so if we update to the latest PATCH version golang in our k8s PATCH versions it will be resolved.
thanks. but In my 3 masters kubernetse clusters. Only one master got this Error . others works well. this confused me. this problem like connection working with ipv6 not ipv4?
you can try asking in the #sig-network channel on slack or log a new issue.

Related discussion:
text: } func (saw *sampleAndWaterMarkHistograms) innerSet(updateXOrX1 func()) { <del> var when time.Time var whenInt int64 var acc sampleAndWaterMarkAccumulator var wellOrdered bool func() { </del> <ins> when, whenInt, acc, wellOrdered := func() (time.Time, int64, sampleAndWaterMarkAccumulator, bool) { </ins> saw.Lock() defer saw.Unlock() <del> when = saw.clock.Now() whenInt = saw.quantize(when) acc = saw.sampleAndWaterMarkAccumulator wellOrdered = !when.Before(acc.lastSet) </del> <ins> // Moved these variables here to tiptoe around https://github.com/golang/go/issues/43570 for #97685 when := saw.clock.Now() whenInt := saw.quantize(when) acc := saw.sampleAndWaterMarkAccumulator wellOrdered := !when.Before(acc.lastSet) </ins> updateXOrX1() saw.relX = saw.x / saw.x1 if wellOrdered {