Code change:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: When I change default hugepage size 2Mi to 1Gi and reboot, kubelet can't update node status What you expected to happen: kubelet can update node info and node status was ready How to reproduce it (as minimally and precisely as possible): change the node condition validate Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration:OS (e.g: ):Kernel (e.g. ):Install tools:Network plugin and version (if this is a network-related bug):Others:
/sig node
Hi, Ref. the error message, it looks like you have allocated hugepages for multiple sizes, . Can you run: to verify? If more than one of the numbers is , that is the reason. This is currently being discussed in , and the reason for implementing the check is here: Imo. this should be supported, and it would only require removing an if statement in order to implement it. I see no reason to limit the node from allocating huge pages with more than one size. Because of how it is implemented, adding support for allowing a pod to request huge pages with multiple sizes isn't trivial, and will require some potentially breaking changes, but that is not relevant to this problem. /cc
Also, can you also provide the result of , that will list the supported huge page sizes (both when is set, and when it is not.).
Thanks ! Looks like your kernel only enables huge pages with the default size. My kernel 5.0+ enable all (atleast 2MiB and 1GiB on x8664) by default. When the node booted with defaulthugepagesz=1GB it "removed support" the 2MiB one, and becuse kubelet use patch on the node status when connecting to the api server, you ended up with both and , even tho 2Mi wast supported by your machine (the sw, not the hw). Even if the error is removed, this will result in a crash when kubelet tries to write to the 2MiB hugetlb cgroup files. I think using should work, otherwise the only current solution is to remove the node from the cluster, and rejoin afterwords. Ref. it looks like you have sorted out the problem, but I still think this is a valid issue, so don't close it. The correct behaviour for kubelet is to remove unsupported sizes, and only allow use of the supported ones.
"Imo. this should be supported, and it would only require removing an if statement in order to implement it. I see no reason to limit the node from allocating huge pages with more than one size." Could you fix this one as well? We use 2M pages only for pods but we need 1G pages for system processes. So it was little surprising when k8s doesn't start because two different pages were configured no matter they are really used or not. Thanks!
I am hitting the same, I used tuned to change the default size from 2Mi to 1Gi, i hit the same problem, after the reboot node is NotReady and it cannot rejoin , giving the error: Node "test-worker-0" is invalid: [status.capacity.hugepages-2Mi: Invalid value: resource.Quantity{i:resource.int64Amount{value:, scale:0}, d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:"20Mi", Format:"BinarySI"}: may not have pre-allocated hugepages for multiple page sizes, status.allocatable.hugepages-2Mi: Invalid value: resource.Quantity{i:resource.int64Amount{value:, scale:0}, My system just reports information for 1Gi, not 2Mi anymore. Removing the node and rebooting fixed it, the node rejoined again reporting the right capacity. But this is not a solution for us, changing sizes should be supported.
Hitting same K8s installed with my ubuntu 18 enabled 2Mi hugepage by default. As I change my master node's hugepage to 1Gi, this happened. I have only one master, so I have to rebuild the k8s cluster.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale
I'm a little confused. when i used tuned to change the hugepagesz from "defaulthugepagesz=2M hugepagesz=2M hugepages=8192" to " defaulthugepagesz=1G hugepagesz=1G hugepages=16", i hit the same problem, after the reboot node is NotReadybut when i used tuned to change the hugepagesz from " defaulthugepagesz=1G hugepagesz=1G hugepages=16" to "default_hugepagesz=2M hugepagesz=2M hugepages=8192", after the rebootkubelet status can still keep Ready. why?
Hi This will be fixed in , but it is still not merged. Hoping to get it into next release. In order to solve the issue (and use 1GiB huge pages) you have to delete the node in kubernetes via , and join the node again.
Thanks
Thanks
I think this is already fixed by can you confirm?
Hi WIth this would not cause a crash, but is still required in order to fix this properly.
I am facing the same issue. After changing the hugepage setting to . I stopped the kubelet process on the worker node and removed the node from the cluster using command and rebooted it. But after it comes back online and kubelet starts, it struggles to join the cluster . Do I have to run the join command again ?

Related discussion:
text: } } <ins> func TestReconcileHugePageResource(t *testing.T) { testKubelet := newTestKubelet(t, false /* controllerAttachDetachEnabled */) hugePageResourceName64Ki := v1.ResourceName("hugepages-64Ki") hugePageResourceName2Mi := v1.ResourceName("hugepages-2Mi") hugePageResourceName1Gi := v1.ResourceName("hugepages-1Gi") cases := []struct { name         string testKubelet  *TestKubelet initialNode  *v1.Node existingNode *v1.Node expectedNode *v1.Node needsUpdate  bool }{ { name:        "no update needed when all huge page resources are similar", testKubelet: testKubelet, needsUpdate: false, initialNode: &v1.Node{ Status: v1.NodeStatus{ Capacity: v1.ResourceList{ v1.ResourceCPU:              *resource.NewMilliQuantity(2000, resource.DecimalSI), v1.ResourceMemory:           *resource.NewQuantity(10e9, resource.BinarySI), v1.ResourceEphemeralStorage: *resource.NewQuantity(5000, resource.BinarySI), hugePageResourceName2Mi:     resource.MustParse("100Mi"), hugePageResourceName64Ki:    *resource.NewQuantity(0, resource.BinarySI), }, Allocatable: v1.ResourceList{ v1.ResourceCPU:              *resource.NewMilliQuantity(2000, resource.DecimalSI), v1.ResourceMemory:           *resource.NewQuantity(10e9, resource.BinarySI), v1.ResourceEphemeralStorage: *resource.NewQuantity(5000, resource.BinarySI), hugePageResourceName2Mi:     resource.MustParse("100Mi"), hugePageResourceName64Ki:    *resource.NewQuantity(0, resource.BinarySI), }, }, }, existingNode: &v1.Node{ Status: v1.NodeStatus{ Capacity: v1.ResourceList{ v1.ResourceCPU:              *resource.NewMilliQuantity(2000, resource.DecimalSI), v1.ResourceMemory:           *resource.NewQuantity(10e9, resource.BinarySI), v1.ResourceEphemeralStorage: *resource.NewQuantity(5000, resource.BinarySI), hugePageResourceName2Mi:     resource.MustParse("100Mi"), hugePageResourceName64Ki:    *resource.NewQuantity(0, resource.BinarySI), }, Allocatable: v1.ResourceList{ v1.ResourceCPU:              *resource.NewMilliQuantity(2000, resource.DecimalSI), v1.ResourceMemory:           *resource.NewQuantity(10e9, resource.BinarySI), v1.ResourceEphemeralStorage: *resource.NewQuantity(5000, resource.BinarySI), hugePageResourceName2Mi:     resource.MustParse("100Mi"), hugePageResourceName64Ki:    *resource.NewQuantity(0, resource.BinarySI), }, }, }, expectedNode: &v1.Node{ Status: v1.NodeStatus{ Capacity: v1.ResourceList{ v1.ResourceCPU:              *resource.NewMilliQuantity(2000, resource.DecimalSI), v1.ResourceMemory:           *resource.NewQuantity(10e9, resource.BinarySI), v1.ResourceEphemeralStorage: *resource.NewQuantity(5000, resource.BinarySI), hugePageResourceName2Mi:     resource.MustParse("100Mi"), hugePageResourceName64Ki:    *resource.NewQuantity(0, resource.BinarySI), }, Allocatable: v1.ResourceList{ v1.ResourceCPU:              *resource.NewMilliQuantity(2000, resource.DecimalSI), v1.ResourceMemory:           *resource.NewQuantity(10e9, resource.BinarySI), v1.ResourceEphemeralStorage: *resource.NewQuantity(5000, resource.BinarySI), hugePageResourceName2Mi:     resource.MustParse("100Mi"), hugePageResourceName64Ki:    *resource.NewQuantity(0, resource.BinarySI), }, }, }, }, { name:        "update needed when new huge page resources is supported", testKubelet: testKubelet, needsUpdate: true, initialNode: &v1.Node{ Status: v1.NodeStatus{ Capacity: v1.ResourceList{ v1.ResourceCPU:              *resource.NewMilliQuantity(2000, resource.DecimalSI), v1.ResourceMemory:           *resource.NewQuantity(10e9, resource.BinarySI), v1.ResourceEphemeralStorage: *resource.NewQuantity(5000, resource.BinarySI), hugePageResourceName2Mi:     *resource.NewQuantity(0, resource.BinarySI), hugePageResourceName1Gi:     resource.MustParse("2Gi"), }, Allocatable: v1.ResourceList{ v1.ResourceCPU:              *resource.NewMilliQuantity(2000, resource.DecimalSI), v1.ResourceMemory:           *resource.NewQuantity(10e9, resource.BinarySI), v1.ResourceEphemeralStorage: *resource.NewQuantity(5000, resource.BinarySI), hugePageResourceName2Mi:     *resource.NewQuantity(0, resource.BinarySI), hugePageResourceName1Gi:     resource.MustParse("2Gi"), }, }, }, existingNode: &v1.Node{ Status: v1.NodeStatus{ Capacity: v1.ResourceList{ v1.ResourceCPU:              *resource.NewMilliQuantity(2000, resource.DecimalSI), v1.ResourceMemory:           *resource.NewQuantity(10e9, resource.BinarySI), v1.ResourceEphemeralStorage: *resource.NewQuantity(5000, resource.BinarySI), hugePageResourceName2Mi:     resource.MustParse("100Mi"), }, Allocatable: v1.ResourceList{ v1.ResourceCPU:              *resource.NewMilliQuantity(2000, resource.DecimalSI), v1.ResourceMemory:           *resource.NewQuantity(10e9, resource.BinarySI), v1.ResourceEphemeralStorage: *resource.NewQuantity(5000, resource.BinarySI), hugePageResourceName2Mi:     resource.MustParse("100Mi"), }, }, }, expectedNode: &v1.Node{ Status: v1.NodeStatus{ Capacity: v1.ResourceList{ v1.ResourceCPU:              *resource.NewMilliQuantity(2000, resource.DecimalSI), v1.ResourceMemory:           *resource.NewQuantity(10e9, resource.BinarySI), v1.ResourceEphemeralStorage: *resource.NewQuantity(5000, resource.BinarySI), hugePageResourceName2Mi:     *resource.NewQuantity(0, resource.BinarySI), hugePageResourceName1Gi:     resource.MustParse("2Gi"), }, Allocatable: v1.ResourceList{ v1.ResourceCPU:              *resource.NewMilliQuantity(2000, resource.DecimalSI), v1.ResourceMemory:           *resource.NewQuantity(10e9, resource.BinarySI), v1.ResourceEphemeralStorage: *resource.NewQuantity(5000, resource.BinarySI), hugePageResourceName2Mi:     *resource.NewQuantity(0, resource.BinarySI), hugePageResourceName1Gi:     resource.MustParse("2Gi"), }, }, }, }, { name:        "update needed when huge page resource quantity has changed", testKubelet: testKubelet, needsUpdate: true, initialNode: &v1.Node{ Status: v1.NodeStatus{ Capacity: v1.ResourceList{ v1.ResourceCPU:              *resource.NewMilliQuantity(2000, resource.DecimalSI), v1.ResourceMemory:           *resource.NewQuantity(10e9, resource.BinarySI), v1.ResourceEphemeralStorage: *resource.NewQuantity(5000, resource.BinarySI), hugePageResourceName1Gi:     resource.MustParse("4Gi"), }, Allocatable: v1.ResourceList{ v1.ResourceCPU:              *resource.NewMilliQuantity(2000, resource.DecimalSI), v1.ResourceMemory:           *resource.NewQuantity(10e9, resource.BinarySI), v1.ResourceEphemeralStorage: *resource.NewQuantity(5000, resource.BinarySI), hugePageResourceName1Gi:     resource.MustParse("4Gi"), }, }, }, existingNode: &v1.Node{ Status: v1.NodeStatus{ Capacity: v1.ResourceList{ v1.ResourceCPU:              *resource.NewMilliQuantity(2000, resource.DecimalSI), v1.ResourceMemory:           *resource.NewQuantity(10e9, resource.BinarySI), v1.ResourceEphemeralStorage: *resource.NewQuantity(5000, resource.BinarySI), hugePageResourceName1Gi:     resource.MustParse("2Gi"), }, Allocatable: v1.ResourceList{ v1.ResourceCPU:              *resource.NewMilliQuantity(2000, resource.DecimalSI), v1.ResourceMemory:           *resource.NewQuantity(10e9, resource.BinarySI), v1.ResourceEphemeralStorage: *resource.NewQuantity(5000, resource.BinarySI), hugePageResourceName1Gi:     resource.MustParse("2Gi"), }, }, }, expectedNode: &v1.Node{ Status: v1.NodeStatus{ Capacity: v1.ResourceList{ v1.ResourceCPU:              *resource.NewMilliQuantity(2000, resource.DecimalSI), v1.ResourceMemory:           *resource.NewQuantity(10e9, resource.BinarySI), v1.ResourceEphemeralStorage: *resource.NewQuantity(5000, resource.BinarySI), hugePageResourceName1Gi:     resource.MustParse("4Gi"), }, Allocatable: v1.ResourceList{ v1.ResourceCPU:              *resource.NewMilliQuantity(2000, resource.DecimalSI), v1.ResourceMemory:           *resource.NewQuantity(10e9, resource.BinarySI), v1.ResourceEphemeralStorage: *resource.NewQuantity(5000, resource.BinarySI), hugePageResourceName1Gi:     resource.MustParse("4Gi"), }, }, }, }, { name:        "update needed when a huge page resources is no longer supported", testKubelet: testKubelet, needsUpdate: true, initialNode: &v1.Node{ Status: v1.NodeStatus{ Capacity: v1.ResourceList{ v1.ResourceCPU:              *resource.NewMilliQuantity(2000, resource.DecimalSI), v1.ResourceMemory:           *resource.NewQuantity(10e9, resource.BinarySI), v1.ResourceEphemeralStorage: *resource.NewQuantity(5000, resource.BinarySI), hugePageResourceName1Gi:     resource.MustParse("2Gi"), }, Allocatable: v1.ResourceList{ v1.ResourceCPU:              *resource.NewMilliQuantity(2000, resource.DecimalSI), v1.ResourceMemory:           *resource.NewQuantity(10e9, resource.BinarySI), v1.ResourceEphemeralStorage: *resource.NewQuantity(5000, resource.BinarySI), hugePageResourceName1Gi:     resource.MustParse("2Gi"), }, }, }, existingNode: &v1.Node{ Status: v1.NodeStatus{ Capacity: v1.ResourceList{ v1.ResourceCPU:              *resource.NewMilliQuantity(2000, resource.DecimalSI), v1.ResourceMemory:           *resource.NewQuantity(10e9, resource.BinarySI), v1.ResourceEphemeralStorage: *resource.NewQuantity(5000, resource.BinarySI), hugePageResourceName2Mi:     *resource.NewQuantity(0, resource.BinarySI), hugePageResourceName1Gi:     resource.MustParse("2Gi"), }, Allocatable: v1.ResourceList{ v1.ResourceCPU:              *resource.NewMilliQuantity(2000, resource.DecimalSI), v1.ResourceMemory:           *resource.NewQuantity(10e9, resource.BinarySI), v1.ResourceEphemeralStorage: *resource.NewQuantity(5000, resource.BinarySI), hugePageResourceName2Mi:     *resource.NewQuantity(0, resource.BinarySI), hugePageResourceName1Gi:     resource.MustParse("2Gi"), }, }, }, expectedNode: &v1.Node{ Status: v1.NodeStatus{ Capacity: v1.ResourceList{ v1.ResourceCPU:              *resource.NewMilliQuantity(2000, resource.DecimalSI), v1.ResourceMemory:           *resource.NewQuantity(10e9, resource.BinarySI), v1.ResourceEphemeralStorage: *resource.NewQuantity(5000, resource.BinarySI), hugePageResourceName1Gi:     resource.MustParse("2Gi"), }, Allocatable: v1.ResourceList{ v1.ResourceCPU:              *resource.NewMilliQuantity(2000, resource.DecimalSI), v1.ResourceMemory:           *resource.NewQuantity(10e9, resource.BinarySI), v1.ResourceEphemeralStorage: *resource.NewQuantity(5000, resource.BinarySI), hugePageResourceName1Gi:     resource.MustParse("2Gi"), }, }, }, }, } for _, tc := range cases { t.Run(tc.name, func(T *testing.T) { defer testKubelet.Cleanup() kubelet := testKubelet.kubelet needsUpdate := kubelet.reconcileHugePageResource(tc.initialNode, tc.existingNode) assert.Equal(t, tc.needsUpdate, needsUpdate, tc.name) assert.Equal(t, tc.expectedNode, tc.existingNode, tc.name) }) } } </ins> func TestReconcileExtendedResource(t *testing.T) { testKubelet := newTestKubelet(t, false /* controllerAttachDetachEnabled */) testKubelet.kubelet.kubeClient = nil // ensure only the heartbeat client is used