jiebi/CodeConvo huggingface dataset:
might have thoughts as well
Based on internal cloud profiling data from thousands of Windows clusters, pdh!PdhCollectQueryData takes 33% CPU usage as of today, 01/20/2023. If the updating interval increases from 1s to 10s, pdh!PdhCollectQueryData usage can be reduced 90% and CPU usage of kubelet can potentially drop 30%.
that is a compelling improvement. I am still concerned about changing the default on folks who might rely on it, I am not sure if it is a valid concern though
To understand the impact of changing the interval, what is the purpose for querying those Windows counters?
- Can you share a little bit more information about your setup? What node size are you using in AKS? Are you overprovisioning your nodes? Do you have any addons running in your AKS cluster? Does the drop eventually go down? Have you seen this issue on Windows Server 2022 too? We don't see this spike in CPU usage in our clusters provisioned with CAPZ (cluster api provider for azure) so I suspect something else is going on too. https://perf- I'm not opposed to increasing the timeout but currently I think this issue may be environmental.
1 The node size is D32_Flex which has 32 vCPUs and 128 GiB memory. 2 The perf counter query usage is the same across all clusters regardless of the cluster is running hot or cold. 3 azurepolicy is the only addon enabled. 4 The trending of perf counter query usage is quite stable. 5 Not applicable at this moment, all clusters are running with Windows Server 2019. Do you have CPU traces from CAPZ to check the time spent on PdhCollectQueryData? For our internal profiling data, perf counter usage is pretty stable in kubelet, and it's not related to the CPU spike.
I am not sure I understand this statement. The cpu usage from the the winstats pkg would be attributed to the kubelet since it is the own process. What is the owning process for usage you are seeing?
usage is from kubelet process itself. One thing I misunderstood is that, the usage in kubelet is not stable. Actually, it keeps growing.
Nodes can't recognize the static pods that they create possibly because the defaulting is different. always returns false. Here is a json diff of two pod specs before and after they are Semenatic.DeepEqual'd where before is created pod and after is what kubelet things the created pod should be. ! Thus: p0 for triage
I think the solution is to not apply defaults to mirror pods in the API Server.
Yes,we have to add specific code to API Server to detect the mirror pod, then not apply defaults.
The right fix should be , which we didn't get to before v1.0. I will start on this and at least get it in for v1.1 to prevent future complications. Since we cannot update the node/kubelet, options are: suggests to disable applying defaults to mirror pods as a temporary fix. IIRC, was against treating mirror pod differently at the apiserver. Since this is only a workaround, perhaps we can relax the constraint? a script to update the pod manifest files.
Disabling defaults has a number of problems: are version-specific. Conversion may fail if defaults are not applied. may fail if defaults are not applied. using defaulted fields assumes default values have been applied. defaulting pass is not the only place where values can be set, as mentioned in . For instance, if the user were to change the kube-system LimitRange, that would cause the same problem. This is another case where would have been useful. Which gives me another idea. For mirror pods, we could capture the entire metadata+spec provided by the user as an annotation, as we do in the client for kubectl apply, and then we could return that in GET. PUT may not work. We'd have to resolve the size concerns raised by . How serious of a problem is this delete-recreate loop?
Ok, we are going with approach proposed in . I am closing this issue as dup of
"entire metadata+spec" in plain text? It'd be hard for kubelet to diff using this. It'd be better to let kubelet store hash of its own choosing so that it can easily compare when there is any change.
Even with , we can't fix v1.0 kubelet.
I agree with
Yes, I were confused.
The comparison is in Kubelet, and we cannot patch 1.0 kubelet in this case.
There are no serious issues with this bug: static pods are running properly, but corresponding mirror pods coming and gone. cc/ Can we treat this as part of GKE cluster upgrade process? Can we upgrade GKE master nodes, then ask the users to upgrade their fluentd yaml file in their cluster? For OSS kubernetes users, most likely they won't run into this issue. But we should document this as known issue for 1.1.
Or can we update the fluentd yaml files when updating the master?
cc/
This is a cc issue, really. But the fast answer is no, because "then ask the users to upgrade their fluentd yaml file in their cluster?" Historically our users are extremely slow to do maintenance.
Our current plan for this issue is just document as a known issue for 1.1 release.
In a 100 node cluster this is going to cause +100 writes +100 deletes per second. Is that extra load a concern?
This would be +100 writes +100 deletes every 10 seconds in a 100-node cluster. We are still discussing options to let user update their manifest files.
Ok, from the logs in the issue description, it looked more frequent but most of the deletes are failing with key not found
's easy to trigger a sync for all pods in kubelet. Any change in any pod would trigger a sync. In this case, the DELETE/ADD of the mirror pod could also trigger another sync. So, yes, it could be worse than 10s... We are looking at patching this in the apiserver, based on suggestion.
Yeah, my suggestion is fixing defaulting/conversion/validation as needed to make exceptions for mirror pods.
I'm not 100% sure it's needed, but if necessary we can turn the defaulting fix on in apiserver, and leave it off in scheduler, so that it gets the defaults it expects.
Yeah, it is way more frequent than 10s. I pointed this out to earlier. We had a meeting with and looks like: Providing a scripts to convert fluentd yaml file and asking the user to run is not a solution. Our users are very slow on such maintenance work. Piggyback such conversion work on master updating might not a solution for GKE clusters.
We do have access to GKE nodes, and we don't do that. To obtain the permission to run script and sync a file requires more work from PM side. This is a API backward compatibility issue anyway. Given today's version management policy, we should handle such compatibility issue. I checked our default code: request diff was introduced by TerminationGracePeriodSeconds was introduced along with graceful termination support. There is no conversion for both fields There is no validation checking both fields today I think it is safe to disable such defaulting for mirror pods shouldn't cause any issue for 1.1 release. Sending PR shortly.
In order to not break the v1.1+ kubelet (which applies newer defaults), we should only skip defaulting for pods from pre-v1.1 kubelets. An easy way to do this is to reuse , which changes the mirror key annotation value from "mirror" to the hash of the pod manifest file. In short, if the key exists, it's a mirror pod. If the value is "mirror", it is sent by a pre-v1.1 kubelet. Otherwise, it's sent by a v1.1+ kubelet. Note that we will also need to maintain the special defaulting case for the future releases until we drop support for v1.0 nodes.
Just this morning, we discussed providing a script/tool to update static pod configs for people with large clusters. Why have we changed that plan? Any hack in apiserver is going to be confusing and fragile. For instance, a LimitRange installed for kube-system would defeat the hack. If we were to add a hack in apiserver, we'd need to remove it in 1.2.
we talked to and the conclusion was that node update may not be feasible as . I agree that any apiserver fix is hacky and fragile, but it seems the most viable option right now. We would need to carry this fix in apiserver as long as GKE supports v1.N master + v1.0 node, unless we can make updating the configs part of the cluster upgrade process.
Ah. These fields are problems because they existed in 1.0, but weren't used nor populated. Entirely new fields are silently dropped during decoding.
We should cherrypick the solution into 1.0.7, since it appears there will be one.
That is, we should cherrypick into the 1.0 branch.
Ref.
If the hack is going to remain indefinitely, I'm not comfortable circumventing defaulting, conversion, and validation.
Just not performing the defaulting for these specific fields on just mirror pods seems safer.
In particular, I could easily imagine a new version of the pod API in 1.2 or 1.3, which will require defaulting and conversion to be applied, in general.
We'll be discussing this soon after 1.1 lands, I expect. My proposal would be for v1.N masters to support v1.N-1 and v1.N-2 (and maybe v1.N-3) nodes. That gives users 6-9 months to lag their nodes behind their masters. But it warrants much further discussion. This isn't a viable solution either, because we can't depend on users updating their nodes to the latest 1.0 rev. We expect arbitrary 1.0 nodes to work, including 1.0.6 all the way back to 1.0.1, per our current .
As Ike noted, cherrypicking into 1.0 kubelets is not feasible for solving this, because the whole reason for the problem is that people aren't updating their nodes. The conversion fix in is much preferable to the defaulting fix: it's less brittle, and components that need the defaulting (e.g. scheduler) will see the defaulting (because they run the conversion stack themselves), but on the wire the mirror pods from old kubelets will remain undefaulted.
I didn't say cherrypicking into 1.0 would solve the problem for everything. The behavior in 1.0 is pretty broken, even when not considering upgrade. We have a fix. We should cherrypick it into 1.0.
I don't see how different components would see different values for these fields, assuming they are all updated to the new code. As discussed offline, the scheduler and node controller need to handle cases of empty requests and pod termination for 1.0 kubelets, anyway, so I think the conversion approach is fine. Please put the explanation in code comments, though, not just in issue/PR comments.
Since the conversion stack does the defaulting at the end and runs in every component, and scheduler, node-controller, etc all consume api.Pod objects (instead of v1.Pod objects), they will have a view of the object with the fields defaulted. The conversion based fix essentially retains the defaulting in *api.Pod objects, but undoes it in any serialized representation of the object. Over the wire the defaulting will not be baked in to the object. Older clients that have not been recompiled (including these kubelets) would see a version of the object without the fields being defaulted.
Defaulting is run first:
Oh, you're right, I somehow had that backwards in my head. Sorry, everyone. I think that tilts me back in favor of . This means that defaulting is part of the versioned structs and therefore changing the defaulting is an API breaking change, which we claimed we wouldn't do. Can we change our minds about the defaulting? How about moving this particular defaulting into conversion functions, which is the intended way of doing something like this?
In , we only reset the defaults when converting api.Pod to v1.Pod. Shouldn't all components still see the defaulting values? Agreed that this is an API breaking change. Perhaps we should not change the defaults at all? mentioned in that scheduler already deals with empty requests. It's true in , but seems untrue in . Maybe this should be fixed. As for , we can also ask kubelet to apply a default value if we truly on to have the 30 second dfault.
resources.requests and terminationGracePeriodSeconds are effectively newly fields, so setting defaults is not a breaking change for most clients. If Kubelet were comparing the pre-decoded pods returned by apisever, it would barf on every new field we've . It's an accident of the implementation that it broke on exactly these fields.
Actually, neither priorities nor predicates takes values from limits any longer. That code is rely on the defaulting pass to copy limits to requests in the case that the requests field is absent. We would need to change that, yes, if we were to not set requests for the 1.0 mirror pods.
This Kubelet behavior was known to be broken back in May: What we're trying to do now is figure out a workaround for this broken behavior.
My recommended solution is: not set resources.requests and terminationGracePeriodSeconds to non-nil values for just 1.0 mirror pods. All other fields and all other pods, including 1.1 mirror pods, should have default values set normally. Doing this in the defaulting pass makes the most sense, assuming that it's straightforward to do given the structure of that pass. the scheduler fit and priority functions to use limits for pods where requests are entirely absent. that the new code in the node controller to handle termination of pods on 1.0 kubelets deals with unset terminationGracePeriodSeconds appropriately. a kind/workaround-removal issue to remove these hacks once we don't need to worry about 1.0 kubelets.
Just for comparison, an alternative: this default setting from the defaulting function. it to the conversion function instead. mirror pods from 1.0 kubelets (MPF1K), store the original value in an annotation. MPF1K, when converting to v1, restore the original value & remove the annotation. No changes needed anywhere else in the system, maintenance cost is very low, we can carry this as long as needed.
I don't see how that keeps the default values off the wire.
Actually, I think I see. Yes, I think that could work.
The defaults are on the v1-internal transition, and removed on the internal-v1 transition.
(3) is converting v1 to IR, and (4) is converting IR to v1, either for storing to etcd or to communicate to another component, such as Kubelet.
I'm saying stop using the default setting code for setting these "breaking" defaults.
Right.
I don't think we want to do this for pods in general, though, just for MPF1K
So I think we can actually just add a few more changes to PR and it should work. (Just move the defaulting.)
Yes, I agree. Just for MPF1K. We can easily broaden it if needed.
SGTM
We're going to want to do something similar for new fields to alpha API versions that we don't want to expose in the stable, serialized version of the API.
So if this works, we should add it to
Yes. Changing defaults is a breaking change and therefore needs to happen in conversion.
Adding new fields with new defaults is fine, just not changing the behavior of existing fields. course I do think kubelet is being a very inflexible client here, but we can't fix the stuff that's already out there.
+1 to both points. We should've patched kubelet sooner to avoid this, but it's out there and kubelet 1.0 remains a stubborn client. Sorry for that. thanks for coming up with the solution! Since both of you agreed on , I will change my PR acoordingly.
So that there is no misunderstanding that leads to future such problems: Whether fields are new or pre-existing, it is an invalid assumption that unset fields may not be changed, whether by defaulting or by some other process, such as admission control.
This issue will be considered fixed for 1.1 once is submitted. (I didn't read this thread carefully enough yet to know whether additional work is necessary before considering this whole issue closed.)
Update: has just been merged. Closing the issue.
Probably we should not close this issue until we do a cherrypick?
The change was cherrypicked, so closing this issue
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug The change in a new field () to the Pod spec that is set to true by default. This is causing problems for DaemonSets in some situations. DaemonSets are using ControllerRevisions to keep track of previous configurations and each revision will contain among other things the Pod spec for the DaemonSet. If a DaemonSet is created with the enableServiceLinks property is set to true, the ControllerRevision will also contain this field. If the cluster is rolled back to a previous version of kubernetes, the DeamonSet struct does not have this field and so the ControllerRevision and the DaemonSet will no longer match. This leads to a new ControllerRevision being created and a rolling update of all pods in the DaemonSet. Would it be a better solution to rename the field to and enable service links whenever the field is not set? This would fix the problem with the DaemonSets unless the field is explicitly set before the rollback. This is causing the master upgrade tests to fail as reported in /sig apps /cc /cc /cc
I guess the change in podSpec and the defaulting will break all the workload resources as they are based on comparing the template's podSpec. ~Is the new value defaulted to true also for the DaemonSet's podSpec template? If not, controller might go hot loop like with mutating admission controllers.~ I don't like us making such changes to v1 podSpec. The other way around seems more sensible. /cc
Why is this a bad thing? We will continue to add fields to podSpec - the issue isn't the defaulting (this would happen regardless of which way it defaulted), it's that the controller can't really be version aware. The new value would default to true in the daemonset as well.
I'm not sure that a rollback requiring daemonsets to recreate is a bug. I think that's potentially "the best possible thing that we can do in that case".
/cc SIG Release blocking dashboard with upgrade test failures - https://k8s-
So the problem isn't the fact that a field has been . It the way in which the field was .
enableServiceLinks is bool pointer that defaults to true. To be backward compatible it should be something like where the new default behavior is to assume that service links are enabled, and the field should be set on an opt out basis to disable the service links. compatiblity As it is if you create a new DaemonSet and then roll back cluster to a prior version, or if you have a failover in a HA setup, you can trigger a simultaneous rolling restart of all DaemonSets you created even though you didn't explicitly configure (and may not even be aware of) the new feature. If we respect backward compatibility and make the feature explicitly opt-out (that is the semantics we are going for here in either ca So the problem isn't the fact that a field has been . It the way in which the field was . enableServiceLinks is bool pointer that defaults to true. To be backward compatible it should be something like where the new default behavior is to assume that service links are enabled, and the field should be set on an opt out basis to disable the service links. compatiblity As it is if you create a new DaemonSet and then roll back cluster to a prior version, or if you have a failover in a HA setup, you can trigger a simultaneous rolling restart of all DaemonSets you created even though you didn't explicitly configure (and may not even be aware of) the new feature. If we respect backward compatibility and make the feature explicitly opt-out (that is the semantics we are going for here in either case), you won't trigger the rolling restart on rollback or controller manager failoverse), you won't trigger the rolling restart on rollback or controller manager failover unless you explicitly opt out of the behavior. I don't see why the former behavior is more desirable than the latter.
For background/context, the choice between enable/disable was initially discussed here:
Are you saying that the issue is that default value is not the same as the Go "zero value" ? The concept of zero value is interesting but it forces APIs to bend into unnatural shapes to respect it. Since forever Kubernetes has explicitly said that we have version-specific default values which ARE NOT governed by zero values, but instead are governed by the API version. It is safe to assume that if a field is documented as having a default in a version, you can rely on that field being populated with that default, with a few exceptions.
We generally ask for boolean fields to be "in the affirmative direction", as in this case. Now, we could revisit this. It would be a significant change in direction, but it's not impossible. But I don't think that solves your problem in general -- if this field were optional string, with a default value of "Hello there", you would be in the same boat right?
So the underlying issue is that we are defaulting into the PodSpec which gets adopted into all of the PodTemplateSpecs of every object. That sort of defaults the users declared intent. Why not just default the value into the Pod itself . Is the intention really to default this in all Templates or to make it optional in the Template and default it in the Pod itself?
/milestone v1.13
I can't speak to the ideology of whether to default the field in all PodSpecs or just the Pod, but your suggestion makes sense to me. I'm not sure of the past reasoning for adding other defaulted fields, most recently SchedulerName in Jan. 2017 (), to all PodSpecs instead of just to the Pod object. Did we not have these same tests in place then? This would effectively be saying we should never add new defaulted fields to PodSpec. If this is the plan, I can create a PR for the fix and I'll add a big warning comment to SetDefaults_PodSpec.
As an example of where we don't default the template we have a CronJob that embeds JobTemplate, we never default that template, only an actual Job gets defaulting. I agree with that not defaulting any templates does make sense.
Agree to only default this in pods. Agree to adding the comment. Most important defaulting happens in pods to keep higher level pod specs simpler. We may need to raise the general pattern in a discussion with api review in general, since this is a change from previous strategy, but the previous strategy was not explicitly chosen (to my recollection at least)
Hi all do we have an agreed path forward for fixing this issue in 1.13? if so can we have an ETA for the fix to land? the release team would like for the tests to get passing in the blocking suites in an effort to unmask any other failures that might show up once these are fixed.
/assign
We have a PR with a fix that will hopefully merge today or tomorrow. I will keep an eye on the dashboard to make sure it fixes the failing daemonset tests.
Thanks Can we link the PR here plz. /cc
Reopening, because of the auto-close. Tests are not passing yet. /reopen
Reopening this issue. <detailsIn response to : Reopening, because of the auto-close. Tests are not passing yet. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
PR that should (hopefully) close this issue is here:
DaemonSet tests passed on run 1842, seems to have been fixed: https://k8s- if it doesn't pop up again - /close
Closing this issue. <detailsIn response to : DaemonSet tests passed on run 1842, seems to have been fixed: https://k8s- Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details