Change:
3 node Kubernetes v1.2.4 cluster with HA scheduler and controller-manager have no additional POD running aside from those in the namespaceruns twice per second. Increasing verbosity shows that this is triggered by Stopping all the schedulers and controller-managers, then restarting only one instance of each without stops from running that often.
That seems to be related to
We should try to filter updates to the proxy when the only change is an annotation that it's not watching. This seems to be an unintended consequence of leveraging endpoints.
I'm going to use this one as the root issue.
Do the frequent restores cause any problems?
Performance wise we've had several issues... they can be computationally expensive the larger the number of routes you add.
It's fairly anecdotal, but during the brief time these restores are running, they seem to consume a fair amount of CPU. Especially on an already very heavily-loaded node, this is undesirable.
We should be able to calculate when a restore is a noop and not run the restore. is anyone currently working in this area?
To really know whether an iptables sync is needed you have to do all the work except actually updating it. Working around this in iptables seems like a bad answer - this is delivering 2 events per second per node. Is that really necessary?
It's one update every two seconds by default and it's configurable. By increasing that period, you make a tradeoff with responsiveness of cluster failover. Conceivably updates in large clusters are much more frequent.
maybe Endpoints was the wrong object to watch? , Mike Danese wrote:
From an API perspective, annotating Endpoints, a collection of pods that implements a service, with which endpoint in the collection is the current master makes sense. I don't think anywhere else makes as much sense except for possibly a seperate resource. It's unfortunate that the kube-proxy cannot handle frequent updates to fields that should be noop on these objects. If we can't fix that generally then it shouldn't be hard to special case a fix for the leader election annotation. I'd still like someone to quantify what the effects of these frequent updates are e.g.: frequency/duration/magnitude of utilization spikes whether that utilization is spent traversing the state data structure in process or spent in fork execs This info would be very helpful in priortizing this.
There are only a few objects that are watched by every single node in the cluster. I know it makes sense to be there, but it really is sort of a pessimal choice. , Mike Danese wrote:
We were in favor of component registration, but that seems to have died.
! As you can see from the output of here (rows highlighted in pink), / are using a non-trivial amount of resources over a given 10 second interval; ~1.5 seconds of CPU time and 42 processes launched. You said you wanted to know what was happening internally, but given the number of processes – there were 14 invocations of and 28 of – it's difficult to know precisely what I should . In any case, for something that is essentially a no-op, this activity seems excessive and undesirable.
So we have a BZ open against too many iptables refreshes on heavily loaded clusters but all the comments are private: . The distillation is it's eating time in the kernel (kernel-commit [net] netfilter: xtables: speed up jump target validation" that speeds up iptables operations somewhat.) However, there are still issues with incessant refreshes + large number of rules.
If someone wants to make the kube-proxy logic smarter in the face of no-op changes, I'm happy to entertain that patch, but I am more concerned thatthis is broadcasting events to every node every couple seconds. , Timothy St. Clair
imo we need to lock on a different object, at 5-10k nodes this is an untenable level of overhead.
The anecdote indicates that the immediate perf issue is caused by iptables-restore. In small clusters we can alleviate this by not restoring noops. I'd like to see the frequency of updates in large clusters in regular state (e.g. pods dieing, deployments in progress). Annotation updates may be a small portion of total endpoints updates. If endpoints updates are happening anyway and they are batched, then moving off of endpoints may not be useful. If these updates are not noops then optimizing out noop iptables-restores will also not be useful. I'd be happy with moving to another resource if we identify a better place for this. I'm also happy to revisit the default config for the leaderelection client. I'd be unhappy with a change that diminishes the clarity of the API on an assumption that it will improve performance. It's possible that the tradeoff is worth it but we need to validate/quantify that assumption.
They are batched, but the watch updates will still occur across the cluster. So the more components we have annotating different endpoints the larger the fan-out. Just the notion of a single annotation could cause 10k watch updates for actors who don't care and then have to filter gives me pause. /cc
given that 3rd party api is now in place, would you have any objections if I were to make a pull to allow locking there?
nope, sounds fine.
Exiting crazy release+replanning, part 1 is available for review, onto part 2.
Third party resources don't have validation or conversions across versions. You should probably consider making this a first class API object to gain those benefits. Otherwise things may get really strange across version migrations.
So the only other object that seems appropriate might be component-status. thoughts?
I definitely think that ComponentStatus is better than Endpoints.
k, making change..
SGTM, but IIRC, component status is not backed by etcd?
I just found that out a second ago :-(
So were kind of in a bit of a conundrum. 3rd party doesn't work, and component status isn't stored in etcd. So I see 2 options: the logic for querying componentstatus into something like the node controller and reflect that data etcd. to revisit a scaled down version of component registration to address add-ons in the kubeadm world? Thoughts? /cc
When last I looked at it, the component stuff was heading in a direction to be a convention on top of services/endpoints. That seems right to me -- it is a way to move away from a special type of endpoint that was rushed in a while ago. That doesn't solve the problem of updating leader information. My initial read (without studying this deeply) is that spamming updates to the endpoints object is the wrong way to do this. Can we only update the endpoints object once a new leader is elected and do the spammy stuff elsewhere? (I think that is what the discussion about third party resources used for this is). At the end of the day, is this something that kubernetes should be involved in? If we had a lock server that everyone could talk to (with a security identity and quotas) then that could be used for leader election. Do we want to turn the api server into a generic lock server too?
Only for integrated components, or add-ons, of kubernetes. If we're going to self host the control plane, it's now an implicit requirement if you want HA, otherwise you're adding another etcd which would need it's own setup+config.
What would be really nice is if we could filter a watch based on a field path and only recieve updates if those fields are updated. We need some place to store the lease record. Every resource in kubernetes is a already a lock.
I don't want this to fall off the rails. So it sounds like Option 2 is thorny, but definitely an issue to address as add-ons/components grow. We need a secure reliable mechanism to register with the system and stating leader information is a pretty natural thing to do with such a facility. In the long run, I still believe it is the correct solution, but I doubt we are going to get there soon. That being said the issue presented here is an immediate concern and I don't see any major down sides to Option 1 listed above, but I'd need to do it in a couple of parts.
xref (the component status PR ) which is lost forever because appearantly chokes on it?
I only just realized this was happening after I think the real cost of kubeproxy restoring rules multiple times a second is increased iptables lock contention (and cpu). It might stall concurrent kubelet iptables modifications for some smallish period. I'm inclined to not put in a spot fix for kubeproxy, but I want to make sure it isn't an issue, ?
Is it a spot fix to not call restore if there is no diff in the rules? How hard is it to calculate this diff? That doesn't resolve this issue but it seems like it would mitigate one of the bad symptoms.
I think it should be easy. Definitely should make 1.5. If no ones impacted by this "hotlooping", cherrypick into 1.4.X will depend on exactly how easy the fix is, if people are impacted, we should fix and cherrypick regardless. I'll need to poke around a bit.
I don't agree with the fix suggested by Basically the problem is that every "Endpoints" object is watched by all KubeProxy in the cluster. Which means, that if we are renewing the lease every 2s (which IIUC is the current case), and we have 2000-node clusters, that means that 2000 KubeProxy components are processing that Endpoint update every 2s. No matter if we will be updating iptables in that case or not, this is clearly a waste of time. For leader election we should use an object that components that run on nodes don't listen to - this in my opinion is pretty important requirement in this dicussion.
So I need "buy-in" on the proposed fix around pushing component status into a etcd-backed resource which it currently isn't. Are there any oppositions to this?
If you want to make it so kubeproxy doesn't get a watch event that it needs to ignore, less work for me :) I was proposing something in the here and now, given that this is happening in live 1.4 clusters. I'm guessing component status is not landing in time for 1.4.2.
yeah - I'm not sure if we even cherrypick it at all. So if we want to fix it in 1.4 branch (do you think we want - I don't know how important it is), then probably what you described should be a temporary solution. - regarding buy-in, I don't have any strong objections about it. Though may have some. WDYT?
I haven't seen this until now. What component is using the endpoint objects to do locking?
I would prefer to see us provide pods a way to present any information currently in componentstatus in their own pod status, and get rid of componentstatus entirely. It's a service we should provide for all components, not just system components.
Short term, assuming the locking behavior is legit, either kube-proxy needs to filter out fields it doesn't care about, or we need to get apiserver to filter out such changes (better but much more complex). If kube-proxy is relying on the system to minimize the number of update messages it gets, that's a mistaken assumption in kube-proxy.
here's another user for your alternate view proposal.
Controller manager and scheduler have done this since 1.2.
That should affect a vanishingly small percentage of endpoints, so not much point to doing the extra work to fix it in apiserver. IMO this is just a plain old kube-proxy bug. , Mike Danese wrote:
kube-proxy can be smarter for sure, but this still broadcasts a useless change to every node every 2 seconds. It really should be a different object that is less likely to be actually used. IMO , Daniel Smith wrote:
+1000 - this is exactly what I was trying to express in
+1000 to and , The fan-out at large scale is simply untenable. So on the scale of a 10k node cluster where we had say a minimum of 2 components being updated every 2 seconds that is 10k x (60/2)*2 = 600,000 broadcasts in a minute even with filtering on the proxy side. I for one am a fan of open transparent communication, but this may be a bit too much ;-).
Ah, now I see what you're talking about. I agree it sounds bad, but is it a big percentage of the endpoints traffic at that scale? This costs 10k broadcasts per second. Assume there's 50 pods per node. Assume each pod shows in one service. Assume a pod changes status once every thousand seconds (most pods won't change often, but pods get started/stopped and crashloop often enough to raise the average). 50  10k  .001 = 500 changes per second, makes 5,000,000 broadcasts per second. So the number you are focusing on is like 1% of the number of endpoint changes at that scale... Maybe my numbers are off above, but are they off by two orders of magnitude? I doubt it. So really, you are running into a scaling problem that's fundamental to endpoints, and none of the above solutions will really address it.
I think there's actually a very old issue about endpoints not scaling, probably knows off the top of his head.
(Note that I don't object to changing scheduler and controller manager to use a different object for locking--endpoints was a weird choice IMO--but it is only adding a few feet onto the end of your runway.)
DO NOT persist componentstatuses in etcd. That API is a disaster. See for discussion of its future directions. I don't think endpoints scalability has its own issue, though it has been mentioned in a bunch of places: Pagination is
Ah, I may be misremembering IRL conversations as github issues... , Brian Grant wrote:
Endpoints as an API object has scalability problems (needs some form of pagination) and kube-proxy should batch updates (and filter no-ops). I am 99% sure both of these have open issues. , Daniel Smith wrote:
I don't see how pagination addresses the fan-out problem, or any problem. Even in a 10k node cluster there's only 500k - 1m pods total, and those will be merged into fewer endpoints objects, maybe 2x to 1000x fewer, depends on how big the services are. I think adding a low-cost endpoints view using the mechanism Clayton proposed is not enough; we can filter out metadata changes, ordering changes, and inactive endpoint changes, which kube-proxy doesn't care about (right?). I think that, at best, that filtering will only be 2-3x reduction, which is not enough. So intelligent update batching will be required, or we need a different distribution mechanism. (kube-proxy binaries could do some sort of p2p endpoint distribution, for example.) , Tim Hockin wrote:
(The right kind of update batching would require extensive changes to our watching model, in case it wasn't clear.)
Pagination is about the 1M etcd limit - at ~16 bytes to store an IPv4 address in JSON we will overflow that object after 32k-64k endpoints in the best case, potentially much less in the worst case. I know that feels like a lot, but I'd rather not get surprised by this one. This is not unique to endpoints. , Daniel Smith wrote:
If you have lots of services then, yes. That said I don't think it's super urgent problem. - do you have the same limit for size of value in etcd v3?
Guys, I'm sorry to jump in here with little context, but we need some solution to this issue in the short term. This is seriously impacting the performance of our nodes in production, and from my perspective it would make sense to add a fix that addresses the immediate issue now (probably in kube-proxy), and then separately make the endpoints API better in the medium term. It would be a shame for this issue to sit for ages while we debate the proper solution. Does this make sense? I'm very happy to pitch in and write some code if necessary
Above volunteered to look into the kube-proxy performance issue. Thanks, Beeps.
How about a configmap?
I was waiting for someone to complain (). Can you clarify what exactly is the issue? () just cpu or are you running something else that needs the iptables lock? kube-proxy should already have a request: 100m, meaning that's what it gets if there's contention, and if there isn't it can take how much it needs.
forked, this issue can be about node local implications of spammy watch notifications for kube-proxy
That's not pagination, that's sharding the endpoint object, which may not be necessary in etcd3 on account of size. I'd sooner worry about write contention with that many endpoints. (Pagination is where apiserver serves a list request in multiple pages.)
I guess my point is more that if kube-proxy / iptables can't handle the 2 second updates without being a CPU hog, it's going to have major problems in even medium sized clusters. Is there something odd about OP's setup? Why aren't other people complaining? Why does kube-proxy update iptables as fast as it can, why not batch and update every 5 seconds or 15? Can it precompute something and then take a lock for less time?
Yeah I think there's kubeproxy optimization and unnecessary watch notifications from overloading an object broadcast cluster wide. That's why i forked the bug.
They're not paying close attention to node performance profiles...? Also, we do HA differently so it never jumped out in our profiles, except during bulk operations. (see comment below) Broadcasting is still an issue, but we've proposed a similar approach:
Yes, the issue is CPU performance. In my you can see the sort of CPU is consuming. On heavily-loaded nodes, this penalty and the associated context switches are not insignificant. I'm not sure why others are not seeing this issue to be honest; we are running HA master, on AWS, with some quite heavily-loaded worker nodes. I don't know how common this is, or as points out, how much attention others are paying to their node performance. It's also possible that we are doing something peculiar, but I can't think what. If you want me to gather more data about the performance impact this has I can, but I think I can say with certainty that this is a real problem.
Hmm, in that comment it looks like you have nothing else on the node. If you're saying it's getting much more than its "request: 100m" even on a loaded node, that sounds like a different bug. Of course we should still fix it, I'm just wary of pointing fingers at something that has a relatively low request.
Agree, there's an open issue for proxy to batch updates. There's really no need for it ever to run more than once per second, and even that is probably too much. , Daniel Smith wrote:
FTR - I did try to to repro this and had kube-proxy writing out thousands of iptables rules as fast as it could, and I never saw a single iptables-restore more than 120ms or so, (numbers from memory) , Prashanth B wrote:
/cc - fyi...
Since nobody mentioned this yet: the sheer amount of logging is causing an issue in itself. Is it really necessary to write out iptables this verbose? Also in journalctl it all shows up in red suggesting it is sent to stderr which made me feel uncomfortable looking at this for the first time. I searched for errors, but there were none.
IIRC you said you had a 3.x something kernel, while all our customers are either using later kernels, or using a highly rebased network stack from later kernels. I was able to reproduce the issues with a 4.7 kernels on Fedora. I haven't tried recent Ubuntu. But in any case, iptables-restore latency is highly kernel dependent it seems.
What verbosity are you running? The normal setting is --v=2 and that does not log all the iptables state. That is --v=3 , leo-baltus wrote:
It sounds more and more like a kernel regression , Tim Hockin wrote:
Ah, running with --v=4. I wasn't aware of this, this is a setup made by kubeadm. Sorry for the noise. Op 25/10/2016 om 11:19:28 -0700, schreef Tim Hockin:
So if the fan-out is the problem, perhaps we want publish/subscribe instead? A message queue?..
cross-linking . We are a bit behind on versions (still on 1.3) and at least there, there are real updates to the default/kubernetes endpoints object. These come from the apiservers each thinking they're the one and resetting the endpoints list. suggests that there is a flag that needs to be set to remedy that but there is no documentation?
I'm trying to troubleshoot networking issues in my cluster by watching iptables packet counts using . It looks like a symptom of this issue is that the packet counters are continuously being reset by reloading iptables. Also, the rule order also gets updated continuously which makes it difficult to focus on one rule. Am I correct in assuming this issue is the cause?
probably, yes. , ravishivt <:

Discussion:
activeEndpoints := make(map[proxy.ServicePortName]bool) // use a map as a set staleConnections := make(map[endpointServicePair]bool) svcPortToInfoMap := make(map[proxy.ServicePortName][]hostPortInfo) <ins> newEndpointsMap := make(map[proxy.ServicePortName][]*endpointsInfo) </ins> // Update endpoints for services. for i := range allEndpoints {

// Flatten the list of current endpoint infos to just a list of ips as strings curEndpointIPs := flattenEndpointsInfo(curEndpoints) if len(curEndpointIPs) != len(newEndpoints) || !slicesEquiv(slice.CopyStrings(curEndpointIPs), newEndpoints) { <ins> glog.V(3).Infof("Setting endpoints for %q to %+v", svcPort, newEndpoints) // Gather stale connections to removed endpoints </ins> removedEndpoints := getRemovedEndpoints(curEndpointIPs, newEndpoints) for _, ep := range removedEndpoints { staleConnections[endpointServicePair{endpoint: ep, servicePortName: svcPort}] = true } <del> glog.V(3).Infof("Setting endpoints for %q to %+v", svcPort, newEndpoints) // Once the set operations using the list of ips are complete, build the list of endpoint infos proxier.endpointsMap[svcPort] = proxier.buildEndpointInfoList(portsToEndpoints[portname], newEndpoints) </del> } <ins> // Once the set operations using the list of ips are complete, build the list of endpoint infos newEndpointsMap[svcPort] = proxier.buildEndpointInfoList(portsToEndpoints[portname], newEndpoints) </ins> activeEndpoints[svcPort] = true } } <del> // Remove endpoints missing from the update. </del> <ins> // Check stale connections against endpoints missing from the update. </ins> for svcPort := range proxier.endpointsMap { if !activeEndpoints[svcPort] { <ins> glog.V(2).Infof("Removing endpoints for %q", svcPort) </ins> // record endpoints of unactive service to stale connections for _, ep := range proxier.endpointsMap[svcPort] { staleConnections[endpointServicePair{endpoint: ep.ip, servicePortName: svcPort}] = true } <del> glog.V(2).Infof("Removing endpoints for %q", svcPort) delete(proxier.endpointsMap, svcPort) </del> } proxier.updateHealthCheckEntries(svcPort.NamespacedName, svcPortToInfoMap[svcPort]) } <del> proxier.syncProxyRules() </del> <ins> if len(newEndpointsMap) != len(proxier.endpointsMap) || !reflect.DeepEqual(newEndpointsMap, proxier.endpointsMap) { proxier.endpointsMap = newEndpointsMap proxier.syncProxyRules() } else { glog.V(4).Infof("Skipping proxy iptables rule sync on endpoint update because nothing changed") } </ins> proxier.deleteEndpointConnections(staleConnections) }