Change:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: I upgraded kube-apiserver on a Raspberry Pi 3 cluster and after it begins adding the http listeners, the cpu usage goes up on all processors to use every available cycle and all attempts to connect to the api via https (6443) or http (8080) time out. Since I host etcd on the controller nodes, it also becomes unresponsive. If I set GOMAXPROCS=1, it does limit that and prevents etcd from timing out, but all attempts to connect to kube-apiserver still time out. Eventually, the initial IP allocation check times out, causing a fatal error. What you expected to happen: It should idle at about 4%. How to reproduce it (as minimally and precisely as possible): On a raspberry pi 3 running Debian jessie (arm64), I run: Anything else we need to know?: Environment: Kubernetes version (use ): 1.10.3 arm64 Cloud provider or hardware configuration: Raspberry Pi 3 OS (e.g. from /etc/os-release):Kernel (e.g. ): Install tools: n/a Others:
I've tried profiling this, but since it doesn't respond to the http interface, I can't get it that way. I've tried hacking in the ability to write profile data to a file, but the os.Exit in the fatal error log causes the defers to get skipped the the profile file to remain empty or have corrupted data if I do get it to write.
/sig api-machinery
Hello, I nearly problem. Do you use etcd v2? If we use etcd v2, we need "--storage-backend=etcd2" options. But api-server logs don't notice to us.
Nope, etcd 3.2.13 per
/cc
Tested 1.10.5 and 1.11.0... still have the same problem.
Actually, 1.11 is not causing the problem. My test method was flawed. 1.10 is still broken but I can upgrade. /close
I'm still having the issue with 1.11, i'ld like to know what was your fix
I don't know. It worked fine for a while then I needed to restart that node for unrelated purposes and it failed again so I'm still having the problem, too. /reopen
I've built a binary with go 1.10 and it still has the same problem, fwiw.
says but the server being tested was reverted back to 3.1.11 () for e2e testing due to performance problems. I'll try that next.
running etcd 3.2.18 : Still have the issue
Ok, bisected this trouble back to . PTAL
Reverted and commit on v1.11.0 and have a on arm64:
Built with go1.11beta1 which contains several changes to math/big specifically to address performance issues with arm64. Since these functions are used with encryption, they have a direct affect on establishing TLS communications. That build allowed the TLS connections to establish within the timeout (10s) whereas they were not before. I'll try extending the timeouts and building with go1.10 again and see if the problem can be worked around without using the beta compiler. Part of this may be the fact that I was using RSA certificates instead of ECDSA. If you are, too, you may be able to work around this problem by regenerating your certificates to use ECDSA.

Discussion:
"k8s.io/apiserver/pkg/storage/value" ) <del> var ( // The short keepalive timeout and interval have been chosen to aggressively // detect a failed etcd server without introducing much overhead. keepaliveTime    = 30 * time.Second keepaliveTimeout = 10 * time.Second // dialTimeout is the timeout for failing to establish a connection. dialTimeout = 10 * time.Second ) </del> <ins> // The short keepalive timeout and interval have been chosen to aggressively // detect a failed etcd server without introducing much overhead. const keepaliveTime = 30 * time.Second const keepaliveTimeout = 10 * time.Second // dialTimeout is the timeout for failing to establish a connection. // It is set to 20 seconds as times shorter than that will cause TLS connections to fail // on heavily loaded arm64 CPUs (issue #64649) const dialTimeout = 20 * time.Second </ins> func newETCD3HealthCheck(c storagebackend.Config) (func() error, error) { // constructing the etcd v3 client blocks and times out if etcd is not available.