jiebi/CodeConvo huggingface dataset:
Code change:
cpuQuery                  = "Processor(_Total)% Processor Time" memoryPrivWorkingSetQuery = "Process(_Total)Working Set - Private" memoryCommittedBytesQuery = "MemoryCommitted Bytes" <del> // Perf counters are updated every second. This is the same as the default cadvisor collection period // see https://github.com/google/cadvisor/blob/master/docs/runtime_options.md#housekeeping perfCounterUpdatePeriod = 1 * time.Second </del> <ins> // Perf counters are updated 10 seconds. This is the same as the default cadvisor housekeeping interval // set at https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/cadvisor/cadvisor_linux.go perfCounterUpdatePeriod = 10 * time.Second </ins> // defaultCachePeriod is the default cache period for each cpuUsage. // This matches with the cadvisor setting and the time interval we use for containers. // see https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/cadvisor/cadvisor_linux.go#L63

Related discussion:
text: might have thoughts as well
Based on internal cloud profiling data from thousands of Windows clusters, pdh!PdhCollectQueryData takes 33% CPU usage as of today, 01/20/2023. If the updating interval increases from 1s to 10s, pdh!PdhCollectQueryData usage can be reduced 90% and CPU usage of kubelet can potentially drop 30%.
that is a compelling improvement. I am still concerned about changing the default on folks who might rely on it, I am not sure if it is a valid concern though
To understand the impact of changing the interval, what is the purpose for querying those Windows counters?
- Can you share a little bit more information about your setup? What node size are you using in AKS? Are you overprovisioning your nodes? Do you have any addons running in your AKS cluster? Does the drop eventually go down? Have you seen this issue on Windows Server 2022 too? We don't see this spike in CPU usage in our clusters provisioned with CAPZ (cluster api provider for azure) so I suspect something else is going on too. https://perf- I'm not opposed to increasing the timeout but currently I think this issue may be environmental.
1 The node size is D32_Flex which has 32 vCPUs and 128 GiB memory. 2 The perf counter query usage is the same across all clusters regardless of the cluster is running hot or cold. 3 azurepolicy is the only addon enabled. 4 The trending of perf counter query usage is quite stable. 5 Not applicable at this moment, all clusters are running with Windows Server 2019. Do you have CPU traces from CAPZ to check the time spent on PdhCollectQueryData? For our internal profiling data, perf counter usage is pretty stable in kubelet, and it's not related to the CPU spike.
I am not sure I understand this statement. The cpu usage from the the winstats pkg would be attributed to the kubelet since it is the own process. What is the owning process for usage you are seeing?
usage is from kubelet process itself. One thing I misunderstood is that, the usage in kubelet is not stable. Actually, it keeps growing.
Code change:
cpuQuery                  = "Processor(_Total)% Processor Time" memoryPrivWorkingSetQuery = "Process(_Total)Working Set - Private" memoryCommittedBytesQuery = "MemoryCommitted Bytes" <del> // Perf counters are updated every second. This is the same as the default cadvisor collection period // see https://github.com/google/cadvisor/blob/master/docs/runtime_options.md#housekeeping perfCounterUpdatePeriod = 1 * time.Second </del> <ins> // Perf counters are updated 10 seconds. This is the same as the default cadvisor housekeeping interval // set at https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/cadvisor/cadvisor_linux.go perfCounterUpdatePeriod = 10 * time.Second </ins> // defaultCachePeriod is the default cache period for each cpuUsage. // This matches with the cadvisor setting and the time interval we use for containers. // see https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/cadvisor/cadvisor_linux.go#L63

Related discussion:
text: The usage is almost zero originally, but it grows up to 30+% cpu usage in kubelet. Looking into it...
The tool wasn't getting the correct data at the first time. It turns out usage in kubelet is growing from 20% to 30+% across all clusters, and the usage is never close to zero. I will investigate the growing cost of . It'd be great if we can increase the query interval to reduce the cost.
Code change:
UID:               types.UID("a671734a-e8e5-11e4-8fde-42010af09327"), ResourceVersion:   "22", CreationTimestamp: parseTimeOrDie("2015-04-22T11:49:36Z"), <del> Annotations: map[string]string{ "kubernetes.io/config.mirror": "mirror", "kubernetes.io/config.source": "file", }, </del> }, Spec: api.PodSpec{ Volumes: []api.Volume{

Related discussion:
text: Nodes can't recognize the static pods that they create possibly because the defaulting is different. always returns false. Here is a json diff of two pod specs before and after they are Semenatic.DeepEqual'd where before is created pod and after is what kubelet things the created pod should be. ! Thus: p0 for triage
I think the solution is to not apply defaults to mirror pods in the API Server.
Yes,we have to add specific code to API Server to detect the mirror pod, then not apply defaults.
The right fix should be , which we didn't get to before v1.0. I will start on this and at least get it in for v1.1 to prevent future complications. Since we cannot update the node/kubelet, options are: suggests to disable applying defaults to mirror pods as a temporary fix. IIRC, was against treating mirror pod differently at the apiserver. Since this is only a workaround, perhaps we can relax the constraint? a script to update the pod manifest files.
Disabling defaults has a number of problems: are version-specific. Conversion may fail if defaults are not applied. may fail if defaults are not applied. using defaulted fields assumes default values have been applied. defaulting pass is not the only place where values can be set, as mentioned in . For instance, if the user were to change the kube-system LimitRange, that would cause the same problem. This is another case where would have been useful. Which gives me another idea. For mirror pods, we could capture the entire metadata+spec provided by the user as an annotation, as we do in the client for kubectl apply, and then we could return that in GET. PUT may not work. We'd have to resolve the size concerns raised by . How serious of a problem is this delete-recreate loop?
Ok, we are going with approach proposed in . I am closing this issue as dup of
"entire metadata+spec" in plain text? It'd be hard for kubelet to diff using this. It'd be better to let kubelet store hash of its own choosing so that it can easily compare when there is any change.
Even with , we can't fix v1.0 kubelet.
I agree with
Yes, I were confused.
Code change:
UID:               types.UID("a671734a-e8e5-11e4-8fde-42010af09327"), ResourceVersion:   "22", CreationTimestamp: parseTimeOrDie("2015-04-22T11:49:36Z"), <del> Annotations: map[string]string{ "kubernetes.io/config.mirror": "mirror", "kubernetes.io/config.source": "file", }, </del> }, Spec: api.PodSpec{ Volumes: []api.Volume{

Related discussion:
text: The comparison is in Kubelet, and we cannot patch 1.0 kubelet in this case.
There are no serious issues with this bug: static pods are running properly, but corresponding mirror pods coming and gone. cc/ Can we treat this as part of GKE cluster upgrade process? Can we upgrade GKE master nodes, then ask the users to upgrade their fluentd yaml file in their cluster? For OSS kubernetes users, most likely they won't run into this issue. But we should document this as known issue for 1.1.
Or can we update the fluentd yaml files when updating the master?
cc/
This is a cc issue, really. But the fast answer is no, because "then ask the users to upgrade their fluentd yaml file in their cluster?" Historically our users are extremely slow to do maintenance.
Our current plan for this issue is just document as a known issue for 1.1 release.
In a 100 node cluster this is going to cause +100 writes +100 deletes per second. Is that extra load a concern?
This would be +100 writes +100 deletes every 10 seconds in a 100-node cluster. We are still discussing options to let user update their manifest files.
Ok, from the logs in the issue description, it looked more frequent but most of the deletes are failing with key not found
's easy to trigger a sync for all pods in kubelet. Any change in any pod would trigger a sync. In this case, the DELETE/ADD of the mirror pod could also trigger another sync. So, yes, it could be worse than 10s... We are looking at patching this in the apiserver, based on suggestion.
Yeah, my suggestion is fixing defaulting/conversion/validation as needed to make exceptions for mirror pods.
I'm not 100% sure it's needed, but if necessary we can turn the defaulting fix on in apiserver, and leave it off in scheduler, so that it gets the defaults it expects.
Yeah, it is way more frequent than 10s. I pointed this out to earlier. We had a meeting with and looks like: Providing a scripts to convert fluentd yaml file and asking the user to run is not a solution. Our users are very slow on such maintenance work. Piggyback such conversion work on master updating might not a solution for GKE clusters.
Code change:
UID:               types.UID("a671734a-e8e5-11e4-8fde-42010af09327"), ResourceVersion:   "22", CreationTimestamp: parseTimeOrDie("2015-04-22T11:49:36Z"), <del> Annotations: map[string]string{ "kubernetes.io/config.mirror": "mirror", "kubernetes.io/config.source": "file", }, </del> }, Spec: api.PodSpec{ Volumes: []api.Volume{

Related discussion:
text: We do have access to GKE nodes, and we don't do that. To obtain the permission to run script and sync a file requires more work from PM side. This is a API backward compatibility issue anyway. Given today's version management policy, we should handle such compatibility issue. I checked our default code: request diff was introduced by TerminationGracePeriodSeconds was introduced along with graceful termination support. There is no conversion for both fields There is no validation checking both fields today I think it is safe to disable such defaulting for mirror pods shouldn't cause any issue for 1.1 release. Sending PR shortly.
In order to not break the v1.1+ kubelet (which applies newer defaults), we should only skip defaulting for pods from pre-v1.1 kubelets. An easy way to do this is to reuse , which changes the mirror key annotation value from "mirror" to the hash of the pod manifest file. In short, if the key exists, it's a mirror pod. If the value is "mirror", it is sent by a pre-v1.1 kubelet. Otherwise, it's sent by a v1.1+ kubelet. Note that we will also need to maintain the special defaulting case for the future releases until we drop support for v1.0 nodes.
Just this morning, we discussed providing a script/tool to update static pod configs for people with large clusters. Why have we changed that plan? Any hack in apiserver is going to be confusing and fragile. For instance, a LimitRange installed for kube-system would defeat the hack. If we were to add a hack in apiserver, we'd need to remove it in 1.2.
we talked to and the conclusion was that node update may not be feasible as . I agree that any apiserver fix is hacky and fragile, but it seems the most viable option right now. We would need to carry this fix in apiserver as long as GKE supports v1.N master + v1.0 node, unless we can make updating the configs part of the cluster upgrade process.
Ah. These fields are problems because they existed in 1.0, but weren't used nor populated. Entirely new fields are silently dropped during decoding.
Code change:
UID:               types.UID("a671734a-e8e5-11e4-8fde-42010af09327"), ResourceVersion:   "22", CreationTimestamp: parseTimeOrDie("2015-04-22T11:49:36Z"), <del> Annotations: map[string]string{ "kubernetes.io/config.mirror": "mirror", "kubernetes.io/config.source": "file", }, </del> }, Spec: api.PodSpec{ Volumes: []api.Volume{

Related discussion:
text: We should cherrypick the solution into 1.0.7, since it appears there will be one.
That is, we should cherrypick into the 1.0 branch.
Ref.
If the hack is going to remain indefinitely, I'm not comfortable circumventing defaulting, conversion, and validation.
Just not performing the defaulting for these specific fields on just mirror pods seems safer.
In particular, I could easily imagine a new version of the pod API in 1.2 or 1.3, which will require defaulting and conversion to be applied, in general.
We'll be discussing this soon after 1.1 lands, I expect. My proposal would be for v1.N masters to support v1.N-1 and v1.N-2 (and maybe v1.N-3) nodes. That gives users 6-9 months to lag their nodes behind their masters. But it warrants much further discussion. This isn't a viable solution either, because we can't depend on users updating their nodes to the latest 1.0 rev. We expect arbitrary 1.0 nodes to work, including 1.0.6 all the way back to 1.0.1, per our current .
As Ike noted, cherrypicking into 1.0 kubelets is not feasible for solving this, because the whole reason for the problem is that people aren't updating their nodes. The conversion fix in is much preferable to the defaulting fix: it's less brittle, and components that need the defaulting (e.g. scheduler) will see the defaulting (because they run the conversion stack themselves), but on the wire the mirror pods from old kubelets will remain undefaulted.
I didn't say cherrypicking into 1.0 would solve the problem for everything. The behavior in 1.0 is pretty broken, even when not considering upgrade. We have a fix. We should cherrypick it into 1.0.
I don't see how different components would see different values for these fields, assuming they are all updated to the new code. As discussed offline, the scheduler and node controller need to handle cases of empty requests and pod termination for 1.0 kubelets, anyway, so I think the conversion approach is fine. Please put the explanation in code comments, though, not just in issue/PR comments.
Code change:
UID:               types.UID("a671734a-e8e5-11e4-8fde-42010af09327"), ResourceVersion:   "22", CreationTimestamp: parseTimeOrDie("2015-04-22T11:49:36Z"), <del> Annotations: map[string]string{ "kubernetes.io/config.mirror": "mirror", "kubernetes.io/config.source": "file", }, </del> }, Spec: api.PodSpec{ Volumes: []api.Volume{

Related discussion:
text: Since the conversion stack does the defaulting at the end and runs in every component, and scheduler, node-controller, etc all consume api.Pod objects (instead of v1.Pod objects), they will have a view of the object with the fields defaulted. The conversion based fix essentially retains the defaulting in *api.Pod objects, but undoes it in any serialized representation of the object. Over the wire the defaulting will not be baked in to the object. Older clients that have not been recompiled (including these kubelets) would see a version of the object without the fields being defaulted.
Defaulting is run first:
Oh, you're right, I somehow had that backwards in my head. Sorry, everyone. I think that tilts me back in favor of . This means that defaulting is part of the versioned structs and therefore changing the defaulting is an API breaking change, which we claimed we wouldn't do. Can we change our minds about the defaulting? How about moving this particular defaulting into conversion functions, which is the intended way of doing something like this?
In , we only reset the defaults when converting api.Pod to v1.Pod. Shouldn't all components still see the defaulting values? Agreed that this is an API breaking change. Perhaps we should not change the defaults at all? mentioned in that scheduler already deals with empty requests. It's true in , but seems untrue in . Maybe this should be fixed. As for , we can also ask kubelet to apply a default value if we truly on to have the 30 second dfault.
resources.requests and terminationGracePeriodSeconds are effectively newly fields, so setting defaults is not a breaking change for most clients. If Kubelet were comparing the pre-decoded pods returned by apisever, it would barf on every new field we've . It's an accident of the implementation that it broke on exactly these fields.
Actually, neither priorities nor predicates takes values from limits any longer. That code is rely on the defaulting pass to copy limits to requests in the case that the requests field is absent. We would need to change that, yes, if we were to not set requests for the 1.0 mirror pods.
This Kubelet behavior was known to be broken back in May: What we're trying to do now is figure out a workaround for this broken behavior.
Code change:
UID:               types.UID("a671734a-e8e5-11e4-8fde-42010af09327"), ResourceVersion:   "22", CreationTimestamp: parseTimeOrDie("2015-04-22T11:49:36Z"), <del> Annotations: map[string]string{ "kubernetes.io/config.mirror": "mirror", "kubernetes.io/config.source": "file", }, </del> }, Spec: api.PodSpec{ Volumes: []api.Volume{

Related discussion:
text: My recommended solution is: not set resources.requests and terminationGracePeriodSeconds to non-nil values for just 1.0 mirror pods. All other fields and all other pods, including 1.1 mirror pods, should have default values set normally. Doing this in the defaulting pass makes the most sense, assuming that it's straightforward to do given the structure of that pass. the scheduler fit and priority functions to use limits for pods where requests are entirely absent. that the new code in the node controller to handle termination of pods on 1.0 kubelets deals with unset terminationGracePeriodSeconds appropriately. a kind/workaround-removal issue to remove these hacks once we don't need to worry about 1.0 kubelets.
Just for comparison, an alternative: this default setting from the defaulting function. it to the conversion function instead. mirror pods from 1.0 kubelets (MPF1K), store the original value in an annotation. MPF1K, when converting to v1, restore the original value & remove the annotation. No changes needed anywhere else in the system, maintenance cost is very low, we can carry this as long as needed.
I don't see how that keeps the default values off the wire.
Actually, I think I see. Yes, I think that could work.
The defaults are on the v1-internal transition, and removed on the internal-v1 transition.
(3) is converting v1 to IR, and (4) is converting IR to v1, either for storing to etcd or to communicate to another component, such as Kubelet.
I'm saying stop using the default setting code for setting these "breaking" defaults.
Right.
I don't think we want to do this for pods in general, though, just for MPF1K
So I think we can actually just add a few more changes to PR and it should work. (Just move the defaulting.)
Yes, I agree. Just for MPF1K. We can easily broaden it if needed.
SGTM
We're going to want to do something similar for new fields to alpha API versions that we don't want to expose in the stable, serialized version of the API.
So if this works, we should add it to
Yes. Changing defaults is a breaking change and therefore needs to happen in conversion.
Code change:
UID:               types.UID("a671734a-e8e5-11e4-8fde-42010af09327"), ResourceVersion:   "22", CreationTimestamp: parseTimeOrDie("2015-04-22T11:49:36Z"), <del> Annotations: map[string]string{ "kubernetes.io/config.mirror": "mirror", "kubernetes.io/config.source": "file", }, </del> }, Spec: api.PodSpec{ Volumes: []api.Volume{

Related discussion:
text: Adding new fields with new defaults is fine, just not changing the behavior of existing fields. course I do think kubelet is being a very inflexible client here, but we can't fix the stuff that's already out there.
+1 to both points. We should've patched kubelet sooner to avoid this, but it's out there and kubelet 1.0 remains a stubborn client. Sorry for that. thanks for coming up with the solution! Since both of you agreed on , I will change my PR acoordingly.
So that there is no misunderstanding that leads to future such problems: Whether fields are new or pre-existing, it is an invalid assumption that unset fields may not be changed, whether by defaulting or by some other process, such as admission control.
This issue will be considered fixed for 1.1 once is submitted. (I didn't read this thread carefully enough yet to know whether additional work is necessary before considering this whole issue closed.)
Update: has just been merged. Closing the issue.
Probably we should not close this issue until we do a cherrypick?
The change was cherrypicked, so closing this issue
Code change:
if obj.SchedulerName == "" { obj.SchedulerName = v1.DefaultSchedulerName } <del> if obj.EnableServiceLinks == nil { enableServiceLinks := v1.DefaultEnableServiceLinks obj.EnableServiceLinks = &enableServiceLinks } </del> } func SetDefaults_Probe(obj *v1.Probe) { if obj.TimeoutSeconds == 0 {

Related discussion:
text: <!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug The change in a new field () to the Pod spec that is set to true by default. This is causing problems for DaemonSets in some situations. DaemonSets are using ControllerRevisions to keep track of previous configurations and each revision will contain among other things the Pod spec for the DaemonSet. If a DaemonSet is created with the enableServiceLinks property is set to true, the ControllerRevision will also contain this field. If the cluster is rolled back to a previous version of kubernetes, the DeamonSet struct does not have this field and so the ControllerRevision and the DaemonSet will no longer match. This leads to a new ControllerRevision being created and a rolling update of all pods in the DaemonSet. Would it be a better solution to rename the field to and enable service links whenever the field is not set? This would fix the problem with the DaemonSets unless the field is explicitly set before the rollback. This is causing the master upgrade tests to fail as reported in /sig apps /cc /cc /cc
I guess the change in podSpec and the defaulting will break all the workload resources as they are based on comparing the template's podSpec. ~Is the new value defaulted to true also for the DaemonSet's podSpec template? If not, controller might go hot loop like with mutating admission controllers.~ I don't like us making such changes to v1 podSpec. The other way around seems more sensible. /cc
Why is this a bad thing? We will continue to add fields to podSpec - the issue isn't the defaulting (this would happen regardless of which way it defaulted), it's that the controller can't really be version aware. The new value would default to true in the daemonset as well.
I'm not sure that a rollback requiring daemonsets to recreate is a bug. I think that's potentially "the best possible thing that we can do in that case".
/cc SIG Release blocking dashboard with upgrade test failures - https://k8s-
So the problem isn't the fact that a field has been . It the way in which the field was .