Change:
<ins> {% set num_cpus = salt['grains.get']('num_cpus', '') %} </ins> { "apiVersion": "v1beta3", "kind": "Pod",

Discussion:
After recent performance fixes we are now not bound by CPU or disk IOPS. Apiserver during density test uses around 80% of CPU. Still we observe high request latency for pods, events, endpoints and sometimes other resources. I was able to narrow this down to slow etcd. Etcd uses up to 80% of CPU. For some reason we see high latency of requests to etcd. Increasing disk (to increase IOPS) or number of cores doesn't help. In order to meet criteria defined in 1.0 roadmap () we have to make it much faster. Do you have some suggestions how to debug this? Any ideas what could be the reason?
a related question: from the etcd code it seems that etcd is exposing some metrics under /metrics handler: However, when accessing this handler I'm getting 404 error (both with 2.0.9 and 2.0.10 etcd versions). Do you know what could be the reason?
You can try with etcd 2.1.0-alpha0, which gives you a better performance, less CPU usage and the /metrics endpoint (though it is still expr).
Thanks! We will give it a try. Does etcd export pprof data so that we can profile it for lock contention?
Thanks - we will try it. So far I downloaded it locally and indeed it has /metrics endpoint. Thanks!
Not really. If you have time, I think the first thing you can do is to switch the default json encoder to ffjson. Do you know how much performance do you actually need in terms of write/read?
currently we are focusing more on latency. The issue that we observed is that when reading from etcd the whole directory (which is ~3000 entries, each entry of size ~2kB), it can take more than 2 seconds (which seems to be a lot).
Thanks for the suggestion. I will take a look into it.
Yea... That is probably caused by slow json encoding... :(
If that's parsing json shouldn't it mean that there is high CPU usage?
This is what you told me. :) So I highly suspect it is caused by encoding after you gave me more context.
So you're saying that this json encoding happens under lock so that we cannot use more CPU because other requests are blocked. Is my understanding correct?

Can you point us where this encoding happens?
To clarify - by I meant . Sorry for not being clear
The encoding happens without holding any lock: You give etcd multiple cores, and it only uses 80% of one? Is this what you are saying? I still do not quite follow actually.
ffjson seems like a great improvement if the 2x benchmark comparisons vs golang json encoding materialize. Thanks! However I can't help noticing that that even ffjson is more than 30x slower than some protobuf implementations (according to the benchmarks published at ) Have you considered adding a protobuf-based API to etcd to boost performance? Obviously in the short term we might be able to simply "throw cores at the problem", but over time I think that this will become expensive, and a protobuf interface might make sense.
Take a look at:
Beautiful :-)
Let me start from the beginning :) In my test setup master is n1-standard-8 (i.e. 8 cores) machine with 2 PD-SSD disks each 200GB. Under high load generated by density tests total cpu utilization is about 50% (i.e. we use on average 4 cores out of 8). There are two biggest CPU consumers: apiserver -usually up to 3 cores etcd -usually up to 0.8 core (never above 1) With this setup we observe high latency in requests to etcd that list directories with 3000 entries each ~2kB. For some reason etcd does not use all the cpu it could. I tried to look if etcd changes GOMAXPROCS. It seems it doesn't: Maybe this is the reason? As a result etcd cannot consume more than 1 core?
BTW, I would be careful not to conflate JSON and HTTP vs Protobuf and RPC. e.g. One can do Protobuf over HTTP without RPC, and still get most of the performance gains. I'll comment further in the etcd RFC.
Yes. You have to give GOMAXPROCS more cores, then etcd can use more.
We plan to use gRPC, which is protobuf over http2 basically. But we might choose protobuf over plain http with a /rpc endpoint in the end. Not sure yet.
cc
Is it just a regular env variable?

I've only changed this using
Yes. It is.
Perfect! I'll give it a try tomorrow.
This improved stats a lot! The only offending ones (high percentiles above 1s) are listing pods and events: I'll look into this today.
(Just to be clear - this is still on 8 core master with fast drives)
I guess 2 core is enough. Also the read latency has very little thing to do with seek latency. Everything in etcd is in memory for the moment.
Regarding disk - out experiments confirm that seek latency doesn't really matter here. I'm not sure what the difference will be between setting GOMAXPROCS e.g. to 2 and to 4 (we should check it).
so for clarity, did you: export GOMAXPROCS= etcd --myparams ? /cc
Yes. For testing I've just hardcoded this. I'll send PR tomorrow that will automatically set this based on the number of cores. One thing to mention that we still don't use all 8 cores. According to some further debugging now the bottle neck is on apiserver side. I'll debug more tomorrow.