jiebi/CodeConvo huggingface dataset:
Code change:
// We check for double because it needs to support at least the cri-o minimum // plus whatever delta between node usages (which could be up to or at least crioMinMemLimit) func nodesAreTooUtilized(cs clientset.Interface, nodeList *v1.NodeList) bool { <ins> nodeNameToPodList := podListForEachNode(cs) </ins> for _, node := range nodeList.Items { <del> _, memFraction, _, memAllocatable := computeCPUMemFraction(cs, node, podRequestedResource) </del> <ins> _, memFraction, _, memAllocatable := computeCPUMemFraction(node, podRequestedResource, nodeNameToPodList[node.Name]) </ins> if float64(memAllocatable)-(memFraction*float64(memAllocatable)) < float64(2*crioMinMemLimit) { return true }

Related discussion:
text: From After , Multi-AZ Clusters tests in 1.19 and 1.20 started timing out as createBalancedPodForNodes is extremely inefficient in scale: for each node (out of 5k), it lists all pods in a cluster serially (each list takes ~5s), which translates to ~7 hours. Optimizing this function by listing pods once and then in memory by Spec.NodeName /kind bug /priority critical-urgent /sig scheduling
/triage accepted
/assign
/reopen Looks that ComputeCPUMemFraction took only 12 minutes, but it stuck later, on It is scheduling one pod per each node: Which is waiting 5s to be ready: 5k nodes * 5s = ~7h
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign Can you do the pod creations in parallel?
Yes, good idea. I'll update to do that
Code change:
} func TestCreate(t *testing.T) { <del> simpleStorage := &SimpleRESTStorage{} </del> <ins> wait := sync.WaitGroup{} wait.Add(1) simpleStorage := &SimpleRESTStorage{ injectedFunction: func(obj runtime.Object) (returnObj runtime.Object, err error) { wait.Wait() return &Simple{}, nil }, } </ins> handler := Handle(map[string]RESTStorage{ "foo": simpleStorage, }, codec, "/prefix/version", selfLinker)

Related discussion:
text: Intermittent failures on Travis.
Hrm, this passes 100/100 locally for me. I'll look at the code and see if I see anything. --brendan
Code change:
// records an event using ref, event msg.  log to glog using prefix, msg, logFn func (m *imageManager) logIt(ref *v1.ObjectReference, eventtype, event, prefix, msg string, logFn func(args ...interface{})) { if ref != nil { <del> m.recorder.Event(events.ToObjectReference(ref), eventtype, event, msg) </del> <ins> m.recorder.Event(ref, eventtype, event, msg) </ins> } else { logFn(fmt.Sprint(prefix, " ", msg)) }

Related discussion:
text: Is this a BUG REPORT or FEATURE REQUEST?: What happened: Now the and has been consistent in by PR . There is no need to convert v1.ObjectReference to clientv1.ObjectReference in . What you expected to happen: Remove this function and cleanup the codes which call this function. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?:
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <bre.g., for API Machinery<br(2) specifying the label manually: <bre.g., for sig/scalability<br<brNote: method (1) will trigger a notification to the team. You can find the team list and label list
Will submit the cleanup after PR merged.
/sig api-machinery
/sub
Code change:
if _, ok = t.objects[gvr][namespacedName]; ok { if replaceExisting { for _, w := range t.getWatches(gvr, ns) { <del> w.Modify(obj) </del> <ins> // To avoid the object from being accidentally modified by watcher w.Modify(obj.DeepCopyObject()) </ins> } t.objects[gvr][namespacedName] = obj return nil

Related discussion:
text: <!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --Hello, Found a potential data race in testing. There is a test If we update it like this (we actually can do it ): If watcher updates received object we get a data race. No data race My suggestion: send copy of object here:
/sig api-machinery
/cc /triage accepted Do you remember if we fixed similar issues before?
Hi ,The code avoiding data race seems to already exist If you commented the code on above, the following test case will pass (because of data race), so I think there is no data race in the code : )
Hi Did you run my test with flag? Of course if I comment there will be data race. But let's take a look at: We store in and do with the same . So, watcher receives that is stored in map. we call we take the same from : Therefore, if watcher modifies received in another goroutine we have data race. Output:
Hi You're right ,thank you for solving my confusion. One solution could be to copy of , I will try to fix it.
Code change:
serviceAccountName: fluentd-gcp-scaler containers: - name: fluentd-gcp-scaler <del> image: gcr.io/google-containers/fluentd-gcp-scaler:0.1 </del> <ins> image: gcr.io/google-containers/fluentd-gcp-scaler:0.2 </ins> command: - /scaler.sh - --ds-name=fluentd-gcp-v3.0.0

Related discussion:
text: Forking from To summarize, here's what we observed: daemonset calls coming every minute from both fluentd-scaler and addon-manager (verified this by individually turning them on/off). Things we need to understand here: Is the daemonset object continuously toggling b/w 2 states? (we know here that it's RV is increasing continuously) If yes, what field(s) in the object are changing? IIRC value of some label/annotation (I think 'UpdatedPodsScheduled') is changing (probably related to 2 below) Also, why/should the fluentd-scaler send any api request if the resources are already set to the right value? pods are getting deleted and recreated by daemonset-controller when the scaler is enabled (as was also seen in ). Why this is happening? One thing to note here - all those delete calls are preceded by calls from respective kubelets (but maybe that's expected). cc /priority critical-urgent /assign
Marked this as release-blocker as at least some part of it seems like a correctness issue (i.e fluentd pods dying and getting recreated). Note also that this is causing increased mem-usage with daemonset-controller (depicted here - )
Thanks for filing this! The only thing that was changing (apart from ds/fluentd-gcp status) was the resource version.
I modified fluentd scaler to use --dry-run in to check whether there would be a change, but this still keeps on updating, even though there is no change. Apparently, the logic inside kubectl is broken...
Btw, addon manager sends tons of PATCH requests for all resources every minute anyway, so even though I will fix fluent-scaler not to send them, it won't solve everything.
Should this issue actually be closed? It was auto-closed by the bot, but nobody has confirmed that the actual issue is fixed.
I'll let to confirm (looks like he'll be back on Monday)
/status in-progress
Yes, we should keep this issue open until we fully understand what's going wrong and fix the root-cause. I believe much of the discussion wrt this is happening under related issue
ACK. In progress ETA: 20/03/2018 Risks: May delay the release if not satisfactorily resolved
Need a number of successive green runs, then close.
Code change:
serviceAccountName: fluentd-gcp-scaler containers: - name: fluentd-gcp-scaler <del> image: gcr.io/google-containers/fluentd-gcp-scaler:0.1 </del> <ins> image: gcr.io/google-containers/fluentd-gcp-scaler:0.2 </ins> command: - /scaler.sh - --ds-name=fluentd-gcp-v3.0.0

Related discussion:
text: So I took a look and seems like the fluentd-scaler version bump (which was supposed to fix) was merged b/w runs and of our . However, I don't see any visible improvement in controller-manager memory usage around that time: !
Also, I'm noticing such lines occasionally (once every 5-10 secs in the start and once every ~1min later on) from of latest run: And back when the fluentd-scaler wasn't doing anything useful (i.e before ), the had such lines only sparingly.
Now it isn't even sending requests (other than GET) to the apiserver. How do you generate that graph? Is it 99th percentile of memory usage over time? Or what exactly? One difference after introducing fluentd-gcp-scaler is that now there is a change occurring on the DaemonSet (once, but still). Also, what test is failing because of this? I'd like to take a look into the logs.
Answering your questions: I fetched the ResourceUsageSummaries for density test from historical runs (e.g ) and then obtained the 99% mem-usage for controller-manager for each run. Yes. More precisely, we're collecting mem-usage from controller-manager once every 1 min (or sth like that) and then taking 99th %ile of the resulting sample. Density and load tests. Where we're checking that mem usage of controller-manager is less than the given threshold (i.e 500MB). Ref: The tests are occasionally failing due to just crossing the threshold. Overall, the mem-usage had seen an increase back during the scaler change (see graph above). The job is this - https://k8s- Inside , you'll find all necessary logs (also a file named which has the resource-usage data).
The fluentd-scaler update eliminated fluentd restarts problem. Verified from controller-manager logs that we no longer see such logs that we were : This is great, and I believe that moves this out of release-blocker as we no longer face problems I mentioned in
That said, and me will continue looking into increase in mem-usage (which is not that serious). We'll start by verifying that turning the scaler off improves mem-usage.
Feel free to override me if you have a good reason :)
I agree that this doesn't sounds to be release-blocker.
Code change:
serviceAccountName: fluentd-gcp-scaler containers: - name: fluentd-gcp-scaler <del> image: gcr.io/google-containers/fluentd-gcp-scaler:0.1 </del> <ins> image: gcr.io/google-containers/fluentd-gcp-scaler:0.2 </ins> command: - /scaler.sh - --ds-name=fluentd-gcp-v3.0.0

Related discussion:
text: But I would like to see follow-ups at least for issues like: does this memory increase come from? what was claimed to be "no-op" changes for daemon set were causing problems [is that issues in kubectl, apiserver, somewhere else?] Something like that may hit us again in the (near?) future if we won't understand it now. The current solution is really volatile [if there is a bug somewhere we shouldn't workaround it on our side, but instead have this fixed] at the cluster start (maybe it's fine to leave as is, but I would like to see a discussion of that and a conscious decision being made here)
Definitely agree those are questions we need to answer.
Yup. 1) is tested in . 2) I believe apiserver sends watch events even when only resource version changes, which causes the daemonset controller to recreate the pods. So either: resource version shouldn't change when there is no change anywhere else in the resource apiserver shouldn't send watch events when only resource version is changing daemonset should check whether watch event it got is actually changing anything 3) Isn't that an inherent part of the way addon manager works? It creates all sorts of resources, some dependent on the other (e.g. cluster roles and cluster role bindings), all at once. If some dependency is not satisfied, it will get fixed during the next run.
This is the code that is supposed to protect us from situation that something must change when we write object to etcd (resource version is not stored in etcd): So there should never be a situation that only resource version is changing. If it is, there is bug and need to be fixed. - FYI Reconciliation is a different thing that creating something in one state only to change that to a different state by a different component. I'm not saying this is for sure wrong approach. But I would like to see a conscious decision process discussing pros-and-cons of that.
That's what I'd expect as well. This I'm not sure if should be correct behaviour as IIUC watchers need to receive every RV update for correctness reasons ( ) Maybe. But are we sure if RV is just increasing without any object change? For creation of addons, yes (that race will likely last only for starting few seconds).
Code change:
serviceAccountName: fluentd-gcp-scaler containers: - name: fluentd-gcp-scaler <del> image: gcr.io/google-containers/fluentd-gcp-scaler:0.1 </del> <ins> image: gcr.io/google-containers/fluentd-gcp-scaler:0.2 </ins> command: - /scaler.sh - --ds-name=fluentd-gcp-v3.0.0

Related discussion:
text: But this is wrt addon changing cluster resources, which can potentially create race at arbitrary time after cluster start (though your fix may have gotten rid of this).
Ok.. Seems like I missed Wojtek's comment above.
So I just checked the ds-controller code and it seems like we're updating the ds status only if this condition is being violated: Which means one of those fields is probably mismatching. I'll try to add some logs and run the test at --v=4 to get more info.
anything is possible, but we do have tests ensuring this works properly (no-op updates do not actually result in an update+resourceVersion bump in etcd):
So I ran some 100-node experiments against with more debug logs and I think I now understand the reason for this. The controller-manager logs and mainly show the following difference: This logs show-up when the scaler is active. And these pod deletions are happening as part of rolling-upgrade of daemonset to newer version (the one having resources set). You can see they're getting deleted one-by-one, which also explains why it takes so long (and so are happening even 30 mins after the cluster creation). And the reason why they're deleted one-by-one is because ds-controller has a rolling-upgrade strategy where by default. Proof: And the increased mem-usage FMU is because of the rolling-upgrade performed by ds-controller (over the prolonged period) - as it's the only difference I observe from logs Further, the reason why the RV was increasing was because of few fields 'NumUnavailable', 'NumAvailable', etc fluctuating while the rolling-upgrade
I spoke offline with and suggested that we should increase maxUnavailable for the fluentd daemonset to a large enough value so that we're not bottlenecked by it. My reasoning is: fluentd is not some user workload which will have some service downtime in case there are many unavailable at once fluentds on nodes are independent of each other (i.e one fluentd shouldn't need to wait for another node's fluentd to come up) we should try to make the scaling happen as fast as possible - otherwise it may be too late before they cope-up with increased log-load in the cluster I'm going to make that change and test it against my PR (thanks for pointing that we can change maxUnavailable directly in ds config).
Code change:
serviceAccountName: fluentd-gcp-scaler containers: - name: fluentd-gcp-scaler <del> image: gcr.io/google-containers/fluentd-gcp-scaler:0.1 </del> <ins> image: gcr.io/google-containers/fluentd-gcp-scaler:0.2 </ins> command: - /scaler.sh - --ds-name=fluentd-gcp-v3.0.0

Related discussion:
text: So increasing maxUnavailable indeed seems to help (ref: ): All the 101 replicas were deleted at once instead of one-by-one and happened pretty much within 2 mins of the original fluentd-ds seen by the controller: We should get that PR into 1.10.
Thanks for confirming this! Did increasing maxUnavailable also reduce c-m memory usage?
So the run was still over the threshold, but I was expecting it as I ran c-m with increased verbosity (which ). That said, I'll observe how the usages are once that PR gets in - and FMU it's still possible to have higher values due to the rolling-upgrade potentially happening during the test.
Is there anything else to do here or should this be closed now?
Let's close it once we're sure there are no more such failures. We may also need to bump the threshold. On Mon, Mar 26, 2018, 9:46 AM Kubernetes Submit Queue < :
So bumping maxUnavailable for fluentd rolling-upgrade in is causing couple of issues: Our latest 5k-node performance failed as part of BeforeSuite where we're checking that ~all kube-system pods are running or ready. The reason is because quite some fluentd pods were killed as part of the rolling-upgrade and we timed out before all those pods could come up Our 100-node performance runs due to high CPU/mem-usage of c-m observed during density test, in cases when the fluentd upgrade happens during the test - which is kind of expected So at this point, we might want to choose among one of the 2 options: Rollback , which comes at the cost of tradeoffs I explained in Increase our CPU/mem thresholds to eliminate those flakes in 100-node job and increase the "max-allowed-not-ready-pods" in kube-system ns to a higher value, so that our 5k-node test doesn't fail in BeforeSuite Thoughts?
TBH, I don't know why it was merged without waiting for opinions... I really vote for reverting that. We don't want this - this may mean making cluster "not fully usable" from user perspective.Which is not really what we want to test.
SGTM. Let's revert it and increase our thresholds for the time-being. The discussion about changing fluentd rolling-upgrades can continue to happen in the background.
Code change:
serviceAccountName: fluentd-gcp-scaler containers: - name: fluentd-gcp-scaler <del> image: gcr.io/google-containers/fluentd-gcp-scaler:0.1 </del> <ins> image: gcr.io/google-containers/fluentd-gcp-scaler:0.2 </ins> command: - /scaler.sh - --ds-name=fluentd-gcp-v3.0.0

Related discussion:
text: [MILESTONENOTIFIER] Milestone Issue Needs Approval Action required: This issue must have the label applied by a SIG maintainer. If the label is not applied within 5 days, the issue will be moved out of the v1.11 milestone. <details<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
This issue is now fixed after my above PRs. We're having continuous for the job
Code change:
func RunEdit(f *cmdutil.Factory, out, errOut io.Writer, cmd *cobra.Command, args []string, options *EditOptions) error { var printer kubectl.ResourcePrinter var ext string <ins> var addHeader bool </ins> switch format := cmdutil.GetFlagString(cmd, "output"); format { case "json": printer = &kubectl.JSONPrinter{} ext = ".json" <ins> addHeader = false </ins> case "yaml": printer = &kubectl.YAMLPrinter{} ext = ".yaml" <ins> addHeader = true </ins> default: return cmdutil.UsageError(cmd, "The flag 'output' must be one of yaml|json") }

Related discussion:
text: Recently I was experimenting with changing for a Golang pre-purposed faux editor when editing resources when I noticed that it couldn't be marshaled. Upon inspecting the output JSON I noticed it had the same comments that the default yaml version has. Valid JSON cannot have comments. Since the comment also gives warning / error information upon saving it must be decided where to put that in the JSON output. Perhaps a new JSON property?
Maybe just not save the comments as part of the json to get rid of the warning, but still should them?
Is this a 1.2 thing? I do not see any errors when running the two commands. I made an edit in one run, and did not make an edit in a separate run. Maybe I don't understand the problem that is seen.
there isn't a kubernetes error that is shown, the issue is that the output JSON that kubernetes generates is actually invalid JSON due to the comments at the top. This caused errors when I was trying to unmarshal that JSON in another project.
Okay. Got it. The editor is being very strict about json having no comments.
It's not only about editor, it's simply part of language specification. Explanation for this: IMO if we want to use format, we should comply with it's standard. Also currently used sign for comments is unusual in javascript world, from which json evolved. In this world comments are marked in "C++ like" style (what in early days of JSON was also supported), i mean using and . So IMO in time of converting into JSON - we should strip such comments (I would prefer that), or they should be in some documented and specific to kubernetes way into JSON object.
Code change:
dns_replicas: '$(echo "$DNS_REPLICAS" | sed -e "s/'/''/g")' dns_server: '$(echo "$DNS_SERVER_IP" | sed -e "s/'/''/g")' dns_domain: '$(echo "$DNS_DOMAIN" | sed -e "s/'/''/g")' <ins> instance_prefix: '$(echo "$INSTANCE_PREFIX" | sed -e "s/'/''/g")' </ins> EOF # Configure the salt-master

Related discussion:
text: reported a variety of failure modes in .
+1
I figured I'd add to this issue rather than create a new one. I attempted to create a vagrant cluster today (dev tree is up to date) with 1 Master and 1 Minion and the cluster creation is stuck in the 'Validating master' loop ( for over 10 minutes in the validating loop ). salt-master and salt-minion logs seem to be empty ( they don't appear to be running, in fact ).
I still have the cluster running in case anybody wants additional information.
/cc please let me know if there is any other information about my vagrant setup that I need to post here.
I have been working to get more stability in this environment over last few days. Can you try again from HEAD after this merges:
Can you also verify that you built a release prior to running cluster/Kube-up?
yes, I started from scratch and built a release before running cluster/kube- - cluster/kube- seems to fail fairly early in case there is no release build available.
It looks like got merged. I'll try again and get back to you.
The vagrant cluster seems to come up ( completes master and minion validation ). However, there is new credentials related error that is showing up at the end of running cluster/kube- :
I'll add one more problem I've encountered. Runing from head:
New to me. Will investigate more tomorrow. Sent from my iPhone
Also new to me :-) - will see if I can reproduce either. Sent from my iPhone
Your credendials and certificates are not working with new environment. Just delete your .kubernetesvagrant* files and recreate the cluster. It would be nice to automate this task, if you tear down the cluster.
I am able to reproduce the 502 Bad Gateway error and am working on determining what changed in order to provide a resolution over the last couple days. Please assign this issue to me.
The issue appears to me that the kube-apiserver is not getting provisioned by salt correctly. Parsing Salt logs to determine why.
Looks like the file is not getting laid down by salt for some reason ;-)
The Jinja template was changed as part of identifying the cluster. Fixing up now, and will send a PR to close this issue.
that was it, thanks!