Change:
"github.com/GoogleCloudPlatform/kubernetes/pkg/master" "github.com/GoogleCloudPlatform/kubernetes/pkg/tools" "github.com/GoogleCloudPlatform/kubernetes/pkg/util" <ins> forked "github.com/GoogleCloudPlatform/kubernetes/third_party/forked/coreos/go-etcd/etcd" </ins> "github.com/coreos/go-etcd/etcd" "github.com/golang/glog"

Discussion:
When evaluating results of I have noticed a very strange behavior of apiserver when running density tests on a 100 node cluster with 30 pods per node. It seems that sometimes we spent more than 40% of CPU time in net.(*netFD).dial function. My suspicion is that it's because after each watch message we terminate old connection to etcd and create a new one. With multiple watchers and many update this can be very expensive.
!
Does the etcd client not reuse HTTP connections? I thought the go http client did that by default?
Actually looking at that chart I see it stuck in write, not dial-- am I blind?
I think it is write... The number on dail is confusing though...
There is a small, but very heavy branch on the left of the chart with "dial" path".
But it shows . It does not show where is the other 22.87s.
The way I read this profile (might wrong though) is that for 28.90s dial was in the stack, but was not the "top" function for 22.87s I tried to verify that etcd client reuses HTTP connection and I think it doesn't. I simple log line here: In empty 100 node cluster it got called ~10k times in ~3.5 min. This gives 45 dials per second Do you have a theory why we create so many connections? After adding cache in this is responsible for a biggest chunk of CPU time.
I also run density test to check how correlated this is with the load.

Results below (first column is number of dials per minute, second column is time): $ sudo docker logs 2&1 uniq -c 1839 13:12 4737 13:13 1942 13:14 1184 13:15 836 13:16 949 13:17 1360 13:18 880 13:19 770 13:20 1127 13:21 682 13:22 13:23 13:24 13:25 13:26 13:27 13:28 13:29 13:30 13:31 13:32 13:33 13:34 13:35 13:36 13:37 13:38 13:39 13:40 13:41 13:42 13:43 13:44 1314 13:45 244 13:46 Performance test started at 13:23 and finished around 13:44. Correlation is quite obvious. Note also that in the peak it was generating 560 dials per second!
I think that we have a victim for it. What we've done is that we changed the Transport in go-etcd client: but setting: MaxIdleConnsPerHost: (from default 2) This speeded up the test by ~35% I think that is some exaggeration, but it definitely proves that increasing this value solves the problem.

My hypothesis why it solves the problem: have 100 opened watches( watching for the same changes (which definitely happens in 100-node cluster) and all of them are from apiserver (a single client) change that is interesting is written to etcd is responding to all of them at the same time those connections become iddle (or most of them) before the new request is send, and thus most of them are closed (we can have only 2 of them opened for a single client) the next requests within a watch we need to create new connections
Number of dials after change described by :+1: ~$ sudo docker logs 2&1 uniq -c 335 14:29 200 14:30 5 14:31 28 14:32 65 14:33 145 14:34 69 14:35 121 14:36 59 14:37 202 14:39 140 14:40 112 14:42 77 14:43 190 14:44 104 14:45 5 14:46 4 14:47 (this includes run of performance test)
Just a silly question: are we allowed to change files under Godeps/ directory? If not what is the best way to fix it?
Call instead.
is probably right. If you only use one transport, there is a maxIdelConn limitation.
thanks ! Assigning to myself - I will prepare a PR for it.
cool, this is great. max idle is probably a little high-- we don't want to use all of apiserver's available FDs for this. Does, say, 500 still get most of the benefits? (I think if we multiplexed watches that might also help a lot, but obviously it's much easier to keep more idle connections.) (If you really had needed to modify a file in the godeps/ directory, the proper way to do that is to send a PR to the downstream repo, and the update the dependency once they accept it.)
I completely agree is too much. I didn't have time to make more experiments today, but my feeling is that 500 or 1000 should also be enough. I will come up with more accurate number on Monday. Yes - but this would be much bigger change.

I don't think it's worth doing now. It seems it's ot needed. We just need to initialize the client properly. I will create a PR for it.