Code change:
Running : cpu usage goes to 2 cores @ 100% What you expected to happen: Way lower cpu usage. Environment: Kubernetes version (use ):Cloud provider or hardware configuration: GKE OS (e.g: ):Kernel (e.g. ): Linux felix-Latitude-7490 5.3.0-62-generic ~18.04.1-Ubuntu SMP Wed Jun 24 16:17:03 UTC 2020 x8664 x8664 x86_64 GNU/Linux
Hey I just did not understood, the cpu goes to 200% in the node or in the place where kubectl is running? Thanks!
/priority awaiting-more-evidence /sig cli
On my laptop running kubectl
Same issue on mac: kubectl version Client Version: version.Info{Major:"1", Minor:"17+", GitVersion:"v1.17.9-eks-", GitCommit:"", GitTreeState:"clean", BuildDate:"2020-07-17T19:00:19Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"darwin/amd64"} Server Version: version.Info{Major:"1", Minor:"16+", GitVersion:"v1.16.13-eks-", GitCommit:"", GitTreeState:"clean", BuildDate:"2020-07-17T18:48:53Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"} PID COMMAND %CPU TIME #TH #WQ #PORTS MEM PURG CMPRS PGRP PPID STATE BOOSTS %CPUME %CPUOTHRS UID FAULTS COW MSGSENT MSGRECV SYSBSD SYSMACH CSW PAGEINS IDLEW POWER INSTRS CYCLES USER #MREGS RPRVT VPRVT VSIZE KPRVT KSHRD kubectl 100.5 03:30.41 32/1 0 43 16M 0B 0B 1322 running *0[1] 0. 0. 274 36 16 + 117 + 3730 266+ 100.5 costel.lupoaie N/A N/A N/A N/A N/A N/A I noticed the problem after upgrade from 1.15. Maybe it't not kubectl itself but the way it communicates with external auth provider like IAM in my case.
oh I see. Thanks I've no access to cloud providers right now to test, but as you've provisioned in EKS using IAM and is using GKE it MIGHT be a thing. Will wait to see if someone else have access to cloud providers and can test this :)
Maybe I am wrong and it's just some busy waiting. In my case I noticed it when draining nodes with many pods that cannot immediately be stopped because it would break the deployments maxUnavailable limits, so kubectl has to wait until it can stop all the pods on the draining node.
In my case also this was triggered when pods couldn't be evicted immediately because of a violation of pdb.
does not lead to high cpu usage in my case, so I suspect it has nothing to do with IAM/GKE auth, but with evictions failing because of pdb (but I am not experienced enough to know kubectl works in the background).
I can confirm 100% (not 200%) CPU core usage for each process. Currently I'm draining four nodes in parallel (4x ), all four are waiting due to PDB's, and all are at 100% CPU most of the time. I'll try with a newer kubectl next time.
OK, so it seems to be related to PDB and Drain. I'll try to take a look into it, try to reproduce with the last supported clients. Thank you for reporting! /assign
Thanks for looking into it! And my apologies, I forgot to comment on this before: Our cluster is deployed on bare metal on premise, so it's probably not EKS og GKE related.
Is that possible you try draining with "disable-eviction" just to make sure PDB or eviction control is causing this? Thanks!
I found a newer kubectl lying around, so it's now v1.18.8 (1.17 didn't include it seems) I couldn't test on the same cluster, but I just drained a host from our staging env with a similar amount of pods to what we run in prod (~85 vs ~100), and I think you're right. CPU usage certainly didn't go above 20% at any point on any core, and that's basically background noise on this host.
So my theory here is that kubectl keeps waiting for the node to drain and this is causing a high CPU usage. I'm also creating a cluster here to reproduce the behavior, and see what part of the code is this being caused
Confirmed here :) Even in last kubectl v1.19.2 the problem happens. It seems also it's putting some pressure in API Server because of the amount of nodes vs PDB. disable-eviction is a "workaround" not desired if using PDB /remove-priority awaiting-more-evidence /priority important-soon
I'm hitting my head in the wall here trying to figure out why the goroutine causes the peak of CPU usage, so anyone with better skills then me in Go, feel free to join :) is the function taking the whole CPU apparently Will keep digging what part of this function is taking the whole CPU. When removing the and making this 'serial' it seems to keep the CPU in low usage but I don't know if this is the desired scenario, as draining a node could take much longer since other Pods may not be using PDB. EDIT: but it also seems that the drain does not goes further because I forgot to change the return/break stuff inside the loop. So I'm thinking tomorrow maybe create a new 'queue' of the 'failed because of PDB' Pods, and deal with them in a separate way.

Related discussion:
text: if err != nil { errors = append(errors, err) } <del> default: </del> } }