Change:
While running performance tests (start 30 pods per node in 50 nodes cluster) I the following problem. replication controller and resize it to 1500. I saw the following logs: I0327 14:45:09. ] Controller my-hostname-density30--d487-11e4-9336-: Found 0 pods out of 1500 I0327 14:45:16. ] Controller my-hostname-density30--d487-11e4-9336-: Found 835 pods out of 1500 I0327 14:45:24. ] Controller my-hostname-density30--d487-11e4-9336-: Found 1500 pods out of 1500 , looking into scheduler logs, the first time when it attempts to schedule any of those pods is related to the log: I0327 13:45:55. 4551 ] About to try and schedule pod my-hostname-density30--d487-11e4-9336-a0481cabf39bfjo6e This is roughly ~45 seconds after the first pod was created. The question is what the scheduler was doing in the meantime - the only logs from that time are: I0327 13:40:52. 4551 ] Unexpected EOF during watch stream event decoding: unexpected EOF E0327 13:40:52. 4551 ] watch of api.Service ended with error: 401: The event in requested index is outdated and cleared (the requested history has been cleared [4053/16]) [ 5052] I0327 13:45:53. 4551 ] Unexpected EOF during watch stream event decoding: unexpected EOF I0327 13:45:53. 4551 ] Unexpected EOF during watch stream event decoding: unexpected EOF E0327 13:45:53. 4551 ] watch of api.Pod ended with error: 401: The event in requested index is outdated and cleared (the requested history has been cleared [/5082]) [1 3988] E0327 13:45:53. 4551 ] watch of api.Service ended with error: 401: The event in requested index is outdated and cleared (the requested history has been cleared [/5082] ) [] I0327 13:45:55. 4551 ] About to try and schedule pod my-hostname-density30--d487-11e4-9336-a0481cabf39bfjo6e A very similar thing happened in Kubelet. During the same test, the first pod was bind to minion "0jxm" at: I0327 13:46:00. 4551 ] Attempting to bind my-hostname-density30--d487-11e4-9336-a0481cabf39b7aukl to e2e-test-wojtekt-minion- However, Kubelet reacted to it 50 seconds later: I0327 13:46:50. 4712 ] No Infra Container for "my-hostname-density30--d487-11e4-9336-a0481cabf39b7aukldefault" found. All containers will be restarted. Again, it seems that nothing was really happening during that 50 seconds. The Kubelet logs from that time are: I0327 13:44:55. 4712 ] GET /healthz: (886.441µs) 0 [[monit/5.4] 127.0.0.1:] E0327 13:46:43. 4712 ] watch of api.Node ended with error: 401: The event in requested index is outdated and cleared (the requested history has been cleared [/7110]) [ ] I0327 13:46:46. 4712 ] Unexpected EOF during watch stream event decoding: unexpected EOF I0327 13:46:46. 4712 ] Unexpected EOF during watch stream event decoding: unexpected EOF E0327 13:46:47. 4712 ] watch of api.Service ended with error: 401: The event in requested index is outdated and cleared (the requested history has been cleared [/6384]) [18 353] E0327 13:46:47. 4712 ] watch of *api.Pod ended with error: 401: The event in requested index is outdated and cleared (the requested history has been cleared [/8087]) [] I0327 13:46:50. 4712 ] No Infra Container for "my-hostname-density30--d487-11e4-9336-a0481cabf39b8hwgvdefault" found. All containers will be restarted. ... [and it works since then]. Investigate what is happening during that 45 seconds periods of "nothing". cc
My feeling is that it might be related to some watch issues. BTW - since I wouldn't be able to work on it today, feel free to pick it up. If noone picks it up, I can start looking into it on Monday.
/sub
I can only reproduce tail latency (). are your performance tests checked in so I can play with them (preferably with cluster setup instructions)? Edit: Oh, did you just run 's soak test?
I've seen similar problem but in my case scheduler never noticed new pods. Investigating... AFAIU has a similar setup to mine: e2e cluster with 50 nodes n1-standard-1, master n1-standard-4. running density test with 30 pods per node.
Yes - I Filip wrote, I'm using Density e2e test, to start 30 pods per node (see also ). Also note that this is not happening all the time - it's just kind of "flakiness"
To isolate the performance kubelet more, I ran a simple test which resized the RC to create pods with set to a specific node. This bypassed the scheduler completely. In short, it took ~30 seconds for 30 pods to be reported running on one node. This includes: RC creates 30 pods Kubelet receives pod updates from watch Kubelet starts containers (I used kubernetes/pause images, so there should be no pulling time). Kubelet reports the status back to the apiserver The cluster size is irrelevant in this case, since only one node is being tested.
I was running a very similar experiments (i.e. 1-node cluster) some time ago and also didn't observe it at all. Thus my feeling is that it might be not related neither to scheduler not kubelet, but probably to etcd and/or apiserver which cannot handle a bigger load.
In fact I think this is related to watch [I think that from some reason some changes come to the component through watch with some significant delay]
I don't think cluster size is irrelevant in this experiment, because all the data is read from apiserver/etcd which performance is strictly related with the size of the cluster. As said we suspect that the problem is somewhere close to watch itself, but we need to dig deeper to understand the problem better.
I've some logging and I can confirm that this problem is not related to the scheduler or kubelet themself - the issue is (as I thought) that some changes come to those component through watch with a significant delay (e.g. 1 minute). I will continue investigating if it's related to watch itself or maybe some high load on apiserver or etcd.
I meant for the kubelet startup time alone, the cluster size was irrelevant (which was the case in my test). As for when kubelet receives the update for the first time, that's a different story :) My test was to simply provide another data point to potentially rule out the possibility that kubelet's performance is the bottleneck.
reported an issue through . After debugging, I think we ran into this issue a lot: I observed both scheduler and kubelet doesn't pick up the work immediately. The only common pattern used by both components are using watcher.
1500 writes to etcd is a lot. It's enough to expire the 1000 change history. This could mean that if scheduler happens to be doing a re-list when the writes start, it won't be able to watch when it's done with the list, so it will have to do another list. We should measure how long a list takes the scheduler. How sure are you that the clocks on your local system and the scheduler are relatively well synchronized?
: thanks for looking into it too - what you observed in is exactly what I also observed for bigger clusters. I didn't observe it in e2e tests, but I also didn't try to find it in e2e tests. But my feeling is that it's much easier to replicate it in bigger clusters. : yes I agree that this generates high load on etcd - I will try to look deeper at how long "listing pods" lasts - thanks for suggestion! Also - I'm pretty sure it's not related to the time skew (I also had this hypothesis at some point but verified that it's not the case)
I tried to debug it deeper and have few more findings about it. I've a bunch of different logs and here is what is happening: (or Kubelet) is "opening" watch for pods. some time (say 3 minutes) nod pods are changing, so there are no writes with pods in etcd. the meantime, a bunch of other writes to etcd is happening (e.g. we're updating NodeStatus once per ~2 seconds). In 50 node cluster, it's clear than more than 1000 writes can happen during that time. new pods are created (by replication controller). However, they are not send through this watch to scheduler [this is something that I still don't understand why] 5 minutes since the watch was opened it expires. So we reopen it (still with the previous resourceVersion). reports an error: "The event in requested index is outdated and cleared","cause":"the requested history has been cleared" that we "start from the beginning": we list all pods, update the internal storage and open watch from the beginning. From that point everything starts to work fine. So the thing that we need to understand is the point 4, i.e. why the changes are not delivered through watch that was opened some time ago and there were more than 1000 other changes in the meantime (because this is the reason of it) (I'm not 100% sure that those 1000 other changes are not a red herring, but this seems to be the most probable cause)
cc/ coreos etcd expert. Xiang, is this the same / similar issue as coreos/etcd/issues/2048? We are running etcd 2.X, and go-etcd client is 0.4.6.
This is the same issue.
I don't think item 4 in his sequence is the same issue...
This has been fixed in etcd2 I think.
and I think it is the same issue with some fixes, but not fully in etcd 2.X, and item 4 is the new symptoms.
This might be a etcd issue or an application issue. We need to dig into this a little bit.
As I understand item 4, it shouldn't happen. I thought we were using etcd 2.03 now? We should try to duplicate the issue with straight etcd commands to see if it's something we're doing wrong with the watch or if we're hitting an etcd bug. Maybe we could add a little script to our integration test to do this test.
seems to imply the limit is still in 2.0 for 4
Doh, I meant 5. Sorry, too many conversations open at once.
Yeah the 1000 window history isn't the issue here, it's that after you start watching, 1000 unrelated changes seem to cause your watch to silently stop triggering on changes you do care about. It was my understanding that this shouldn't happen. If you watch for change N and etcd sees change N and sees that you don't trigger on it, it should increment your N automatically; so that if you start out in the watch window, you'll stay in the watch window.
(Increasing the 1000 window history might be nice but should be an optimization change, not a correctness change.)
Yes, didn't read the symptoms in 4 carefully enough. ----- Original Message -----
You actually mean 6,right? 5 is fixed in etcd. 4 is wired. 6 is the 1000 window thing.
What do you mean by wired? Is that a typo for weird?
Yea... Sorry for typo... We need to dig into that one.
a suggestion, you could add some tracking around here: I'm curious where in the pipeline the watch events were dropped (etcd-apiserver-reflector-scheduler). I'd planned to do this but just haven't got around to setting up a 50node cluster yet.
You don't need 50 nodes cluster to reproduce the issue. One can easily reproduce this with e2e tests with 2-node-cluster. See my comment at (comment)
I will continue investigating it tomorrow, but if you have any suggestions (like the one from just let me know.
/sub
This is now a fairly serious issue for us, and we need a mitigation ASAP. I actually a theory as to why e2es might be affected: If nothing else, we tend to churn through a fair number of pods fairly quickly. I suspect one thing that happens is that we end of recycling a lot of containers, too, which could be eventful. We seem to hit a lot of extra instability in our e2e after landed, even though that test itself isn't failing, which lines up well with this theory. (One obvious mitigation is to roll that back for e2es, but I'm not happy with that answer.)
(That commit a mini-density test to the e2es.)
I will write an integration test to try and repro this hypothesized etcd bug.
See . Sounds like the next step here is to put some logging into our etcd watch handling code to try and figure out where the delay is. I'd start at the place mentions above. Log timestamps and compare. Perhaps this delay is getting introduced in apiserver? It does have to watch quite a bit of extra stuff to satisfy label/field selectors. That's what I'd rule out first.
If you think that's necessary, I'm fine with that [the easiest fix for it is to add [Skipped] even if count <= 30 in: But this is just a temporary hack - we need to really fix this issue, not just remove a test that is causing it. But if you think that's necessary, I'm happy to do it.
as an update - since this density test doesn't give much value now and you have a hypothesis that this is causing problems in jenkins, I'm switching it off (until this issue is fixed) in
it seems that the "density" test wasn't not responsible for e2e failures - it is switched off since few hours and tests are still failing
Hmm. Something caused instability to spike yesterday, and the PR timing was about right - I was struggling to come up with a reasonable PR at the exact moment the yellow dots started, but that PR landed at an aborted build, then we had a green, then yellows forever. Thanks for testing my theory, at least! Anyone have any other candidates? Something from Monday's commits is either causing a different bug or making this bug worse. On Apr 1, 2015 6:02 AM, "Wojciech Tyczynski" wrote:
when this patch is complete, im thinking we should commit renablement of the density tests as part of the commit?
Regarding the main issue: it seems that the problem is in etcd. I did a bunch of experiments with additional logs and it, In particular, I've logs in go- (in Godeps), where the http requests are sent to etcd to monitor when they are sent and when they are finally processed. Few example logs: observed 27 requests as following (all of them were equal, sent on behalf of different watch channels): keys/registry/pods?recursive=true&wait=true&waitIndex= All of them were sent within 0.02s (between 13:03:31: and 13:03:31:) 13:04:29:: the first pod was "POST-ed" to the apiserver: I0401 13:04:29. 4056 ] POST /api/v1beta1/pods?namespace=default: (14.737101ms) 201 [[kube-controller-manager/v0.14.0 (linux/amd64) kubernetes/] 127.0.0.1:] , the answers for those request were pretty slow: the first one came at 13:06:33: the last one came at 13:08:30: So since the creation of the first pod until the first of the interesting watches was responded by etcd there was more than 2 minutes. On the other hand, in my previous experiments I've seen strong correlation between the time when the first pod was delivered to scheduler and the time when watch exceeded 5 minutes of being alive. From the logs it's definitely not that clear. [update: maybe that's related to to the fact watch is broke by etcd after 5 minute, and etcd simply wasn't responding to anything within 5 minutes?] However, it seems to be related to the load on etcd so I'm not sure why it happened also in e2e tests (where the load is pretty low). I don't have clear ideas what we can do to improve it. What I observed is that there is a huuuuuge load of "Node" objects on etcd (both in terms of writing and reading them). So I think that we should try to decrease it: the time how often node status is updated are a lot of "watch"es open on Nodes (in particular each Kubelet has a watch on its node). However, all those watch channels (although in most of the cases are interested only in a single node), are watching all nodes and after receiving it from etcd are filtering them based on labels. This seems very inefficient - we should try to change is so that each Kubelet is watching only it's own Node. What else can we do? What do you think about those two?
----- Original Message ----- We support per resource watch filtering in many places, where the REST implementation should transform "name" into a special watch on just the affected key. If that's not happening for Nodes it's a bug in the Node REST logic.
I have a feeling that we're doing it only for services: I can't see a similar logic anywhere else, in particular here: The function that is responsible for filtering is applied in etcdWatcher after getting the response from etcd. Should we use the similar logic to field.Selector as in WatchServices (the only allowed field is name)? BTW, I'm happy to work on it.
cc
cc/ (I don't know Jay's github account name. Please forward this to him) This is the related performance issue I mentioned over our hangout. I suspect the issue related to watcher, but haven't completely identify the root cause is etcd server itself or the way we use watcher yet.
yes - we should probably support it in generic However, that means that we will have to field name conversions defined for those objects, and we should be using "" internally.
hi mine is jayunit100 ! :)
I'm going to try and reproduce result in a test.
Still working on this test-- have found a data race so far but I think that's unrelated.
fyi literally this for nodes:
OK so I have a test that at least demonstrates some sort of bug (, second commit). If I watch a bunch of pods, make unrelated changes to events, then make changes to the pods, it works as expected (usually). If however, the unrelated changes are made to pods, somehow I get an etcd error 401. My working theory for how this happens is that the apiserver's filtering step is too slow processing an update and 1000 (!) changes sneak in by the time it starts the watch up again.
I think we're in different time zones so I'm not going to steal this issue from you. Also headed home for the day. I think next step is to figure out the apiserver path from watch pods to etcd (might be this, need to check: ) and figure out what's going on there.
you're right, and it does the filtering through a predicate function: It would be nice to know the ratio of etcd updates that result in a watch event from the apiserver (so times when that match function returns true) : updates that don't. There's also decorator time, which should be negligable because we only run it over matching objects.
We do have plan to totally solve the "1000" problem in v3 api (repeatable read, keep all histories until user compaction, watch progress notification). For v2, we can mitigate the problem if you need by increasing the window to another order (10K).
- Yup we see the same characteristic behavior, the term we've coined in this scenario was "lumpiness" where operations will speed ahead then grind to a standstill. We also see this during draining. FWIW - We also noticed the large number of Node messages being passed around and their overall performance. The larger the cluster the greater the load on the apiserver due to chatter.
thanks for pointing to this: I'm going to do the same for nodes I've already walked through the path from watch pods to etcd yesterday - what I observed was related to etcd. However it was quite challenging because of huge amount of logs there and I might have missed it. Thus, I'm probably going to start with adding support for watching only a single node (as suggested) to make the logs slightly more transparent)
I think your theory makes sense but it's completely different from what I observe in logs (so maybe there are two unrelated issues here). I've run the experiments one more time and got exactly the same result as yesterday (few times): , watch to etcd is translated into a chain of "request-response" http communication - for more details see: 've extended the logging so that I'm logging: a) every time I send the request (from the above place) to etcd b) every time I get the reply for that request from etcd c) in scheduler, every time it initiates the watch (every 5 minutes) the initial resourceVersion for the request. I observed is that the http request to etcd (responsible for watch on all pods) is hanging for 5 minutes without any response. However, in the meantime apiserver sent to etcd 1500 creations of pods (it seems all of them successed at etcd level from the logs). However any of them triggered the watch [New comparing to yesterday] After 5 minutes, the watch is stopped and we are trying to establish a new one. However our resourceVersion is too old, so we get "too old" error from etcd. This triggers "listing pods" (which works fine) and since now scheduler is already aware of all the pods and starts scheduling. So the above suggests that it's clearly related to etcd (it's not the slowness of apiserver, because the response from etcd doesn't even reach the apiserver). To be clear - I tried to replicate this problem using command-line: etcd & etcdctl and completely failed - the only way I can reproduce this particular problem is starting 1500 pods (using replication controller) on 50 nodes cluster. And with that setup is pretty well reproducable. Also - I don't think that changing the number of events stored in etcd to 10k will change anything. And to be clear - I still don't understand the problem. I though it might be related to the pretty high load on etcd, but on the other hand, etcd is responding to PUTs/GETs from apiserver pretty fast (within a second on very high percentiles), but for those "watch" requests we don't get any response for minutes. And thus so far I don't have any idea how to proceed with it...
One more observation: to replicate the problem, I'm starting the test ~2.5 minutes after the watch was opened in the scheduler. If it's started within ~1.5 minutes after starting the watch (which is started every ~5 minutes) it usually works fine. This also suggests that isn't not related to the apiserver load, because it is processing the same amount of data no matter when it starts.
Can you try to isolate the problem? If you highly suspect it's an etcd issue, I can try to locate the problem. But I want to know roughly how can I simulate the case and reproduce it without k8s.
For example. how many keys? read/write rate besides watching? how many watchers? what is the watching distribution? what is the write rate that triggers watch?
In the "healthy situation" we have number of keys: 2000-3000 for most of the time ~30 reads and ~30 writes per second altogether 150-200 watchers assume that there is exactly one watcher that is interesting for us, and basically no writes trigger it Then we start the real test: we start the test ~2.5 minutes after the watch was opened (this is important here and allows to replicate in almost all runs) we generate a huge burst (say 3000 writes) very quickly and write it to etcd as soon as possible all of those writes are interesting for our watch however, none of those are delivered for more than a minute My observation: it's strictly related to how long the watch is opened (since the watch was opened, there were more than 1000 un-interesting writes) if I generate the burst right after the watch was opened, I'm basically not able to replicate the problem Does it help?
[EDIT: ackk github didn't show me all the replies above this is in response to having kubelets watch their own node object] I don't see how that can help. kubelet watches all pods that match a particular host. We have no index on that, so there's no way to make that watch only a single etcd resource. Likewise scheduler watches all unassigned pods, and all assigned pods. Plausibly the thing that would help is to reimplement watch inside apiserver and have only one outgoing watch to etcd (that watches everything). But that's an enormous amount of work.
I am going to reproduce it today or tomorrow. I will let you the result.
thanks! sorry I don't understand your response - for nodes it's totally different than for pods - for pods the answer whether it's assigned or not is based on some field inside pod. However for node it's based on it's name (which is translatable to key) so it's doable to filter it.
One more thing that might be relevant to description of my experiment: those 30-40 writes per second are triggering ~50 watchers (each of those write trigger each of those watchers).
Thanks for this detail.
kubelets watch pods to determine which pods they ought to be running--that's the watch that is the limiting factor on pod performance. Kubelets also watch the node object, but that's just so they can post their status. So we were talking about different watches.
with 50 nodes updating their status' every 2 seconds, it will reduce the kubelet<-apiserver traffic (and I don't see why it should watch all nodes anyway). I believe was complaining that things were too noisy.
Yes - we are probably talking about different watches. I was talking about the second one. And it was introducing a huge churn in my 50-node cluster. But it should be fixed with
I made a small tool at and tried to simulate the case you described. I created a 3 member cluster with etcd 2.0.8. let 200 watchers watching on . let 50 watchers watching on send 50 writes/second to trigger normal watchers wait for 3 minutes sends a lot of writes (thousands/second) to trigger the stress watchers I expect the events to be lost However they are delivered. Did I do something wrong here? Or do you have some suggestion to reproduce it?
We're running etcd 2.0.5. Probably best to try and repro on the same version. Q
There is no huge difference between 2.0.5 and 2.0.8 about this issue. But I will try later.
this seems very similar to our test. I will think tomorrow why this works (or why our setup doesn't work). Thanks a lot for making it!
You can run the stress test on your env pretty easily too. I am happy to dig into this more.
see hmmm... I wonder if your tool would be more effective with a call to runtime.GOMAXPROCS() so that it can parallelize more?
The load I generated is probably higher than described... I can try with more load too.
As of a build this afternoon , the previously mentioned lags are far worse. Schedule lag 2 minutes Draining lag 5+
I think you told me before and I forgot, but what are you doing when you measure those delays? Can you patch in and see if that brings those times down a bit? What does "Draining lag" mean?
I think I finally have a way to replicate the issue at the etcd level. Here is the code that I used in my experiments: [it's highly based on code from yesterday - there are some small differences]. [In the code I've exported it's hardcoded resourceVersion - I was changing it every time before starting the experiment] What I run is: run -action watch -c 50 -prefix "normal" run -action write -c 50 -prefix "normal" [And it seems to be enough to show the problem]. To explain the test case: 'm running 50 writers, each of them is writing it's own object every second (50 writes per second) 'm running 50 watchers, each of them is watching all the of those 50 objects. no event is delivered to some watcher for 5 minutes, there is a "panic". I've run it 3 times, and in all runs I was able reproduce it within 10 minutes, so it seems quite reliable. My feeling is that we are suffering for exactly this problem in Kubernetes. And my current thinking is that it might be some kind of starvation. But this is just my guess. Can you please take a look into it? I'm happy to help you - just let me know what i can do.
BTW - I left the above experiment for 2 hours and after that time only 4 watchers were receiving anything - the other 46 were just hanging forever. So I'm no longer that sure it it's starvation.
I am looking into it. Thanks for helping me reproduce it.
BTW, can you tell me the env you were running it? version of etcd and hash of etcd-client?
The good news is I have reproduced it with your instruction. Just want to make sure I see the same thing.
- I used official 2.0 etcd release (single instance).
I fixed it here
One more thing to be noticed, the streaming watcher still might be canceled when the load is high. It will reach
Thanks a lot for fixing it. Regarding canceling - yes we are aware of it. There is already a code that handles it. One more thing - IIUC in order to this change be effective, we need a new release of etcd, right? So when can we expect the new release?
Yeah getting dropped due to high load should be handled correctly already.
Yes. I will have a release early next week. You can try with the current master though.
Can you take responsibility for testing the new version of etcd (once it is released) to verify that it fixes this problem? You should also roll back 's workaround ()
Sure - will take care of it.
Can you please ping this thread/me when the new etcd release will be built? Thanks!
since the fix the very simple, can we have a patch release here?
We are doing it. We will release it later today or early tomorrow.
Please let us know as it is ready. We can rebuild our docker image with the release and start the test again.
We just cut the etcd v2.0.9: It contains the bug fix for watcher issue.
Great. I will build our docker image for this, you can pick up tonight PCT for your performance validation. Thanks
You can upgrade your cluster with 2.0.9 and validate the fix now.
Will do it today. Thanks for creating the image.
I wasn't able to reproduce this problem with the new version of etcd (it was very easy to reproduce with the previous version). So I'm going to revert the temporary workaround done by and close this issue.
Yay!

Discussion:
package config import ( <del> "time" </del> "github.com/GoogleCloudPlatform/kubernetes/pkg/api" "github.com/GoogleCloudPlatform/kubernetes/pkg/client" "github.com/GoogleCloudPlatform/kubernetes/pkg/client/cache"

} updates <- kubelet.PodUpdate{pods, kubelet.SET, kubelet.ApiserverSource} } <del> // TODO: the 30 second poll loop is here to mitigate #6059 and // shouldn't be neeeded once that is resolved. cache.NewReflector(lw, &api.Pod{}, cache.NewUndeltaStore(send, cache.MetaNamespaceKeyFunc), 30*time.Second).Run() </del> <ins> cache.NewReflector(lw, &api.Pod{}, cache.NewUndeltaStore(send, cache.MetaNamespaceKeyFunc), 0).Run() </ins> } func getHostFieldLabel(apiVersion string) string {

} // Watch and queue pods that need scheduling. <del> // TODO: the 30 second poll loop is here to mitigate #6059 and // shouldn't be neeeded once that is resolved. cache.NewReflector(f.createUnassignedPodLW(), &api.Pod{}, f.PodQueue, 30*time.Second).Run() </del> <ins> cache.NewReflector(f.createUnassignedPodLW(), &api.Pod{}, f.PodQueue, 0).Run() </ins> // Pass through all events to the scheduled pod store, but on a deletion, // also remove from the assumed pods.