Change:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: I wrote a simple driver program () that creates ConfigMap objects as fast as it can, and ran it like this: . In the CSV file produced, two of the creates recorded an error that simply says "timeout": Here is an excerpt from the api-server's log: A few minutes later I checked that my etcd cluster is healthy; it was: I also looked at the on each host for etcd, and each reported that etcd had been running for over a day. What you expected to happen: I expected no such problem, or at least a more informative error message. How to reproduce it (as minimally and precisely as possible): That test driver is pretty minimal. Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: SoftLayer bare metal machines, SSD storage on the etcd servers OS (e.g. from /etc/os-release): Ubuntu 16.04.4 LTS Kernel (e.g. ): Linux mjs-api-dal10-b 4.4.0-122-generic -Ubuntu SMP Mon Apr 23 15:34:04 UTC 2018 x8664 x8664 x86_64 GNU/Linux Install tools: kubeadm driven by an Ansible playbook I wrote, based on Others:
That's weird, I'm not aware of a 10s timeout anywhere (there is a 60s one). does the etcd client have such a timeout?
FYI:
- can you try etcd without SSL/TLS? it could be ( https://k8s-*&repos=kubernetes )
Yes, I will try that, the next time it is convenient to tear everything down and start over.
OK, I tried again with etcd NOT using TLS (neither client/server nor server/server). I still got a "timeout" error. The driver at commit .
I am running kube release 1.10.3 now. I turned v to 5 in the api-server driven by the client, and ran another trial. Here are the errors logged by the driver (with timestamps in UTC): I found this in the stderr of the api-server (the timestamp seems to be in UTC):
The etcd cluster should have no trouble with that write rate.
Around that time, the etcd server that was loaded by the kube api-server logged the following (in US CDT):
try tweaking and ?
I to the loaded api-server's command line, and still got 10s timeouts.
The doc string for seems to say it is irrelevant for create operations.
However, making the client specify a timeout of 17 sec (see ) warded off the problem --- according to my first and only trial so far.
However, my second trial was much more interesting. The api-server logged an error on create --- but the client did not! And the subsequent update and delete operations on the object in question got 404s. Here is what the driver logged about that object: Here is what the api-server logged around that time: The etcd leader logged nothing unusual around that time.
has suggested the mysterious timeout might be due to . I do indeed have the admission controller enabled. However, I have no ResourceQuota objects in any namespace.
I tried removing from the admission controller list, and that made the mysterious 10 second timeouts go away. Why is the ResourceQuota admission controller causing these 10 second timeouts? Remember, they are rare: I can do hundreds of thousands of create operations in rapid fire, and get a single digit number of timeouts, while the other create operations complete in well under 10 seconds. See for an example.
Here is another example (now with and the etcd cluster using TLS): <img width="578" alt="screen shot 2018-05-25 at 1 29 51 pm" src="https://user-"
So I slowed down from 3000 to 2000 kube writes per second, but I still got a timeout. <img width="576" alt="screen shot 2018-05-25 at 2 13 59 pm" src="https://user-"And no, that is not a really tight cluster of data points at (17:35:25.870, 10.036) --- it really is just a single data point. No other operation took anywhere near that long.
Interestingly, there is a roughly 50 ms bubble of latency around the time of the start of the operation that timed out. As you can see from the graph, there are other bubbles of latency. All much smaller than 10 sec. Here are 36 consecutive lines of driver data from around the timeout, sorted by preop time.
Looking in , I see something odd. In two places --- and in --- there is a special case for an empty list of objects. Now, it is not easy to imagine how the list could ever be empty when the namespace is in the queue --- but if this ever comes to pass then something bad happens: is called twice (first in and second in ).
So I made to test the hypothesis that the double-call of is a critical part of the pathology. I ran a few trials, and got no timeouts. Here is the longest:
wanna file a WIP PR so we can see the exact changes?
yeah, when I get a chance. Weekend now.
very nice investigation you've done here!

Discussion:
e.inProgress.Delete(ns) } <ins> // getWork returns a namespace, a list of work items in that // namespace, and a shutdown boolean.  If not shutdown then the return // must eventually be followed by a call on completeWork for the // returned namespace (regardless of whether the work item list is // empty). </ins> func (e *quotaEvaluator) getWork() (string, []*admissionWaiter, bool) { uncastNS, shutdown := e.queue.Get() if shutdown {

work := e.work[ns] delete(e.work, ns) delete(e.dirtyWork, ns) <del> if len(work) != 0 { e.inProgress.Insert(ns) return ns, work, false } e.queue.Done(ns) e.inProgress.Delete(ns) return ns, []*admissionWaiter{}, false </del> <ins> e.inProgress.Insert(ns) return ns, work, false </ins> } // prettyPrint formats a resource list for usage in errors