Change:
Forking from To summarize, here's what we observed: daemonset calls coming every minute from both fluentd-scaler and addon-manager (verified this by individually turning them on/off). Things we need to understand here: Is the daemonset object continuously toggling b/w 2 states? (we know here that it's RV is increasing continuously) If yes, what field(s) in the object are changing? IIRC value of some label/annotation (I think 'UpdatedPodsScheduled') is changing (probably related to 2 below) Also, why/should the fluentd-scaler send any api request if the resources are already set to the right value? pods are getting deleted and recreated by daemonset-controller when the scaler is enabled (as was also seen in ). Why this is happening? One thing to note here - all those delete calls are preceded by calls from respective kubelets (but maybe that's expected). cc /priority critical-urgent /assign
Marked this as release-blocker as at least some part of it seems like a correctness issue (i.e fluentd pods dying and getting recreated). Note also that this is causing increased mem-usage with daemonset-controller (depicted here - )
Thanks for filing this! The only thing that was changing (apart from ds/fluentd-gcp status) was the resource version.
I modified fluentd scaler to use --dry-run in to check whether there would be a change, but this still keeps on updating, even though there is no change. Apparently, the logic inside kubectl is broken...
Btw, addon manager sends tons of PATCH requests for all resources every minute anyway, so even though I will fix fluent-scaler not to send them, it won't solve everything.
Should this issue actually be closed? It was auto-closed by the bot, but nobody has confirmed that the actual issue is fixed.
I'll let to confirm (looks like he'll be back on Monday)
/status in-progress
Yes, we should keep this issue open until we fully understand what's going wrong and fix the root-cause. I believe much of the discussion wrt this is happening under related issue
ACK. In progress ETA: 20/03/2018 Risks: May delay the release if not satisfactorily resolved
Need a number of successive green runs, then close.
So I took a look and seems like the fluentd-scaler version bump (which was supposed to fix) was merged b/w runs and of our . However, I don't see any visible improvement in controller-manager memory usage around that time: !
Also, I'm noticing such lines occasionally (once every 5-10 secs in the start and once every ~1min later on) from of latest run: And back when the fluentd-scaler wasn't doing anything useful (i.e before ), the had such lines only sparingly.
Now it isn't even sending requests (other than GET) to the apiserver. How do you generate that graph? Is it 99th percentile of memory usage over time? Or what exactly? One difference after introducing fluentd-gcp-scaler is that now there is a change occurring on the DaemonSet (once, but still). Also, what test is failing because of this? I'd like to take a look into the logs.
Answering your questions: I fetched the ResourceUsageSummaries for density test from historical runs (e.g ) and then obtained the 99% mem-usage for controller-manager for each run. Yes. More precisely, we're collecting mem-usage from controller-manager once every 1 min (or sth like that) and then taking 99th %ile of the resulting sample. Density and load tests. Where we're checking that mem usage of controller-manager is less than the given threshold (i.e 500MB). Ref: The tests are occasionally failing due to just crossing the threshold. Overall, the mem-usage had seen an increase back during the scaler change (see graph above). The job is this - https://k8s- Inside , you'll find all necessary logs (also a file named which has the resource-usage data).
The fluentd-scaler update eliminated fluentd restarts problem. Verified from controller-manager logs that we no longer see such logs that we were : This is great, and I believe that moves this out of release-blocker as we no longer face problems I mentioned in
That said, and me will continue looking into increase in mem-usage (which is not that serious). We'll start by verifying that turning the scaler off improves mem-usage.
Feel free to override me if you have a good reason :)
I agree that this doesn't sounds to be release-blocker. But I would like to see follow-ups at least for issues like: does this memory increase come from? what was claimed to be "no-op" changes for daemon set were causing problems [is that issues in kubectl, apiserver, somewhere else?] Something like that may hit us again in the (near?) future if we won't understand it now. The current solution is really volatile [if there is a bug somewhere we shouldn't workaround it on our side, but instead have this fixed] at the cluster start (maybe it's fine to leave as is, but I would like to see a discussion of that and a conscious decision being made here)
Definitely agree those are questions we need to answer.
Yup. 1) is tested in . 2) I believe apiserver sends watch events even when only resource version changes, which causes the daemonset controller to recreate the pods. So either: resource version shouldn't change when there is no change anywhere else in the resource apiserver shouldn't send watch events when only resource version is changing daemonset should check whether watch event it got is actually changing anything 3) Isn't that an inherent part of the way addon manager works? It creates all sorts of resources, some dependent on the other (e.g. cluster roles and cluster role bindings), all at once. If some dependency is not satisfied, it will get fixed during the next run.
This is the code that is supposed to protect us from situation that something must change when we write object to etcd (resource version is not stored in etcd): So there should never be a situation that only resource version is changing. If it is, there is bug and need to be fixed. - FYI Reconciliation is a different thing that creating something in one state only to change that to a different state by a different component. I'm not saying this is for sure wrong approach. But I would like to see a conscious decision process discussing pros-and-cons of that.
That's what I'd expect as well. This I'm not sure if should be correct behaviour as IIUC watchers need to receive every RV update for correctness reasons ( ) Maybe. But are we sure if RV is just increasing without any object change? For creation of addons, yes (that race will likely last only for starting few seconds). But this is wrt addon changing cluster resources, which can potentially create race at arbitrary time after cluster start (though your fix may have gotten rid of this).
Ok.. Seems like I missed Wojtek's comment above.
So I just checked the ds-controller code and it seems like we're updating the ds status only if this condition is being violated: Which means one of those fields is probably mismatching. I'll try to add some logs and run the test at --v=4 to get more info.
anything is possible, but we do have tests ensuring this works properly (no-op updates do not actually result in an update+resourceVersion bump in etcd):
So I ran some 100-node experiments against with more debug logs and I think I now understand the reason for this. The controller-manager logs and mainly show the following difference: This logs show-up when the scaler is active. And these pod deletions are happening as part of rolling-upgrade of daemonset to newer version (the one having resources set). You can see they're getting deleted one-by-one, which also explains why it takes so long (and so are happening even 30 mins after the cluster creation). And the reason why they're deleted one-by-one is because ds-controller has a rolling-upgrade strategy where by default. Proof: And the increased mem-usage FMU is because of the rolling-upgrade performed by ds-controller (over the prolonged period) - as it's the only difference I observe from logs Further, the reason why the RV was increasing was because of few fields 'NumUnavailable', 'NumAvailable', etc fluctuating while the rolling-upgrade
I spoke offline with and suggested that we should increase maxUnavailable for the fluentd daemonset to a large enough value so that we're not bottlenecked by it. My reasoning is: fluentd is not some user workload which will have some service downtime in case there are many unavailable at once fluentds on nodes are independent of each other (i.e one fluentd shouldn't need to wait for another node's fluentd to come up) we should try to make the scaling happen as fast as possible - otherwise it may be too late before they cope-up with increased log-load in the cluster I'm going to make that change and test it against my PR (thanks for pointing that we can change maxUnavailable directly in ds config).
So increasing maxUnavailable indeed seems to help (ref: ): All the 101 replicas were deleted at once instead of one-by-one and happened pretty much within 2 mins of the original fluentd-ds seen by the controller: We should get that PR into 1.10.
Thanks for confirming this! Did increasing maxUnavailable also reduce c-m memory usage?
So the run was still over the threshold, but I was expecting it as I ran c-m with increased verbosity (which ). That said, I'll observe how the usages are once that PR gets in - and FMU it's still possible to have higher values due to the rolling-upgrade potentially happening during the test.
Is there anything else to do here or should this be closed now?
Let's close it once we're sure there are no more such failures. We may also need to bump the threshold. On Mon, Mar 26, 2018, 9:46 AM Kubernetes Submit Queue < :
So bumping maxUnavailable for fluentd rolling-upgrade in is causing couple of issues: Our latest 5k-node performance failed as part of BeforeSuite where we're checking that ~all kube-system pods are running or ready. The reason is because quite some fluentd pods were killed as part of the rolling-upgrade and we timed out before all those pods could come up Our 100-node performance runs due to high CPU/mem-usage of c-m observed during density test, in cases when the fluentd upgrade happens during the test - which is kind of expected So at this point, we might want to choose among one of the 2 options: Rollback , which comes at the cost of tradeoffs I explained in Increase our CPU/mem thresholds to eliminate those flakes in 100-node job and increase the "max-allowed-not-ready-pods" in kube-system ns to a higher value, so that our 5k-node test doesn't fail in BeforeSuite Thoughts?
TBH, I don't know why it was merged without waiting for opinions... I really vote for reverting that. We don't want this - this may mean making cluster "not fully usable" from user perspective.Which is not really what we want to test.
SGTM. Let's revert it and increase our thresholds for the time-being. The discussion about changing fluentd rolling-upgrades can continue to happen in the background.
[MILESTONENOTIFIER] Milestone Issue Needs Approval Action required: This issue must have the label applied by a SIG maintainer. If the label is not applied within 5 days, the issue will be moved out of the v1.11 milestone. <details<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
This issue is now fixed after my above PRs. We're having continuous for the job

Discussion:
namespace: kube-system labels: k8s-app: fluentd-gcp-scaler <del> version: v0.1.0 </del> <ins> version: v0.2.0 </ins> addonmanager.kubernetes.io/mode: Reconcile spec: selector:

serviceAccountName: fluentd-gcp-scaler containers: - name: fluentd-gcp-scaler <del> image: gcr.io/google-containers/fluentd-gcp-scaler:0.1 </del> <ins> image: gcr.io/google-containers/fluentd-gcp-scaler:0.2 </ins> command: - /scaler.sh - --ds-name=fluentd-gcp-v3.0.0