Change:
What happened: We ran the following, and got the following: How/why do we have a node reporting 100% memory usage? (There seems to be plenty of memory on the host as reported by the kernel's statistic. (multiple gigabytes) What you expected to happen: Memory usage can't exceed 100%, no? How to reproduce it (as minimally and precisely as possible): we unfortunately don't know Anything else we need to know?: no swap on these VMs. We're curious what kernel memory statistic goes into computing the total from Kubernetes; it's my understanding that there is various ways to go over 100%, e.g., by summing RSS over several processes. (E.g., shared and resident pages would get double-counted.) Environment: Kubernetes version (use ):Cloud provider or hardware configuration: Azure OS (e.g: ): AKS instance? Kernel (e.g. ): (unsure, since this is Azure AKS; we don't have good access to this piece of data…) Install tools: Azure? Network plugin and version (if this is a network-related bug): N/A / Azure Others: None
(Taking my best shot here, but if this is wrong please freely adjust it.) /sig instrumentation
Other than 101% reporting issue, do you observe any other abnormality ?
Well, yes, the node didn't seem to be actually at 100% memory use; as mentioned, it seemed to have significant headroom.
how about
Node Memory utilization is ratio of node Working Set Bytes and node allocatable memory. Allocatable memory is available on node object: Node Working Set Bytes are available on api: Please provide results from those commands so we can distinquish if problem is in kubectl top or in metrics pipeline.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I have similar problem: kubectl describe node secondary-pool-dqiqzzikb5 kubectl get --raw I have no idea what's causing it or how to fix :/
You can't reopen an issue/PR unless you authored it or you are a collaborator. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen We have the same issue.
Reopened this issue. <detailsIn response to : /reopen We have the same issue. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-lifecycle rotten /triage accepted /assign
Seeing same issue here too
Another data point (on AWS EKS): Seems like happen only on smaller nodetypes (I see it with both micro/nano, others are medium) Usage is displayed correctly with NodeMetrics:
Why kubelet node memory utilization can exceed 100% (same can happen for CPU)? Simple, because it's not utilization of physical device, but utilization of resources allocated for pods and system daemons. How utilization of resources allocated is calculated? Simple, it's sum of resources used by on node divided by all resources allocated. How Kubelet allocates resources for pods? It takes all resources available on VM and substracts resources that it reserves for itself, kernel etc. How Kubelet knows how much resources it needs to reserve? User provides it in flags. There is no error, there is no Kubernetes magic that gets us over 100%, it's just question of how you define utilization. Kubelet reserves some resources for system, they are not included when calculating node utilization, which means it can go above 100% when pods start using reserved resources (this is what over-committing means in "kubectl describe node" output). Some math based on your kubectl describe node output Node capacity: 968424Ki Node allocatable: 559848Ki Node Usage 717664Ki Node utilization: Node usage / node allocatable = 717664Ki / 559848Ki = 128% VM utilization: Node usage / node capacity = 717664Ki / 968424Ki = 74%
Based on my reading of the , I don't think "Node Usage" include only usage from pods. The includes usage by system daemons, so I'm not sure it makes sense to compare to allocatable, which is meant to exclude system daemons.
Hmm, I was not aware of that. This is a good point, as utilization was implemented in kubectl by different SIG (CLI), it's possible that it was not properly reviewed by other stakeholders (SIG Instrumentation/SIG Node). I would make sense to revisit what should be displayed as node utilization.

Discussion:
SortBy             string NoHeaders          bool UseProtocolBuffers bool <ins> ShowCapacity       bool </ins> NodeClient      corev1client.CoreV1Interface Printer         *metricsutil.TopCmdPrinter

cmd.Flags().StringVar(&o.SortBy, "sort-by", o.SortBy, "If non-empty, sort nodes list using specified field. The field can be either 'cpu' or 'memory'.") cmd.Flags().BoolVar(&o.NoHeaders, "no-headers", o.NoHeaders, "If present, print output without headers") cmd.Flags().BoolVar(&o.UseProtocolBuffers, "use-protocol-buffers", o.UseProtocolBuffers, "Enables using protocol-buffers to access Metrics API.") <ins> cmd.Flags().BoolVar(&o.ShowCapacity, "show-capacity", o.ShowCapacity, "Print node resources based on Capacity instead of Allocatable(default) of the nodes.") </ins> return cmd }

nodes = append(nodes, nodeList.Items...) } <del> allocatable := make(map[string]v1.ResourceList) </del> <ins> availableResources := make(map[string]v1.ResourceList) </ins> for _, n := range nodes { <del> allocatable[n.Name] = n.Status.Allocatable </del> <ins> if !o.ShowCapacity { availableResources[n.Name] = n.Status.Allocatable } else { availableResources[n.Name] = n.Status.Capacity } </ins> } <del> return o.Printer.PrintNodeMetrics(metrics.Items, allocatable, o.NoHeaders, o.SortBy) </del> <ins> return o.Printer.PrintNodeMetrics(metrics.Items, availableResources, o.NoHeaders, o.SortBy) </ins> } func getNodeMetricsFromMetricsAPI(metricsClient metricsclientset.Interface, resourceName string, selector labels.Selector) (*metricsapi.NodeMetricsList, error) {