Change:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: scheduler doesn't assign pod because 'not enough resource: hugepages-1Gi',but the node info show it has enough resource. describe node What you expected to happen: the pod was in normal status like pending and running How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: It was change default hugepage size to 1Gi and hugepages=36 numastat was show there has enough hugepage also proc/meminfo was show there has enough resource other info Environment: Kubernetes version (use ):Cloud provider or hardware configuration:OS (e.g: ):Kernel (e.g. ):Install tools:Network plugin and version (if this is a network-related bug):Others:
/sig node
cc do you have thoughts here? I see huge pages and think of you :)
Hmm. Are you sure that there are no other pods using huge page memory ? To me, it looks like the node has (huge page @ 1GiB) memory, you are requsting , but is already used. Can you verify that by listing the pods on the node, and check if some of them are using huge page memory already?
Also, the OS will report that no huge pages are in use as long as no processes have started using the hugepages. Eg. when a pod requesting with is assigned to a node, k8s will then mark the s with huge page memory it as "used". The cgroup controller (if hugetlb cgroup is mounted) will then set the usage limit to . Before the pod actually starts using the huge page memory, the OS will report that no huge page memory is used. When a process inside the pod maps with huge page memory via eg. , the OS will see that the memory is in used. If the process tries to map more than the requested amount, the map will fail (given that the hugetlb cgroup is mounted). To verify if the hugetlb cgroup is mounted, run . If mounted, it will retrun something like this:
I’m so sorry that there actually has a other pod using hugepagei only see node info available hugepage. so i close the issuethanks a lot
No problem I think this issue is a symptom that something should be done to make it easier to understand. I will look into it.

Discussion:
corev1.ResourceMemory, memoryReqs.String(), int64(fractionMemoryReqs), memoryLimits.String(), int64(fractionMemoryLimits)) w.Write(LEVEL_1, "%st%s (%d%%)t%s (%d%%)n", corev1.ResourceEphemeralStorage, ephemeralstorageReqs.String(), int64(fractionEphemeralStorageReqs), ephemeralstorageLimits.String(), int64(fractionEphemeralStorageLimits)) extResources := make([]string, 0, len(allocatable)) <ins> hugePageResources := make([]string, 0, len(allocatable)) </ins> for resource := range allocatable { <del> if !resourcehelper.IsStandardContainerResourceName(string(resource)) && resource != corev1.ResourcePods { </del> <ins> if resourcehelper.IsHugePageResourceName(resource) { hugePageResources = append(hugePageResources, string(resource)) } else if !resourcehelper.IsStandardContainerResourceName(string(resource)) && resource != corev1.ResourcePods { </ins> extResources = append(extResources, string(resource)) } } sort.Strings(extResources) <ins> sort.Strings(hugePageResources) for _, resource := range hugePageResources { hugePageSizeRequests, hugePageSizeLimits, hugePageSizeAllocable := reqs[corev1.ResourceName(resource)], limits[corev1.ResourceName(resource)], allocatable[corev1.ResourceName(resource)] fractionHugePageSizeRequests := float64(0) fractionHugePageSizeLimits := float64(0) if hugePageSizeAllocable.Value() != 0 { fractionHugePageSizeRequests = float64(hugePageSizeRequests.Value()) / float64(hugePageSizeAllocable.Value()) * 100 fractionHugePageSizeLimits = float64(hugePageSizeLimits.Value()) / float64(hugePageSizeAllocable.Value()) * 100 } w.Write(LEVEL_1, "%st%s (%d%%)t%s (%d%%)n", resource, hugePageSizeRequests.String(), int64(fractionHugePageSizeRequests), hugePageSizeLimits.String(), int64(fractionHugePageSizeLimits)) } </ins> for _, ext := range extResources { extRequests, extLimits := reqs[corev1.ResourceName(ext)], limits[corev1.ResourceName(ext)] w.Write(LEVEL_1, "%st%st%sn", ext, extRequests.String(), extLimits.String())

} } <ins> func getHugePageResourceList(pageSize, value string) corev1.ResourceList { res := corev1.ResourceList{} if pageSize != "" && value != "" { res[corev1.ResourceName(corev1.ResourceHugePagesPrefix+pageSize)] = resource.MustParse(value) } return res } // mergeResourceLists will merge resoure lists. When two lists have the same resourece, the value from // the last list will be present in the result func mergeResourceLists(resourceLists ...corev1.ResourceList) corev1.ResourceList { result := corev1.ResourceList{} for _, rl := range resourceLists { for resource, quantity := range rl { result[resource] = quantity } } return result } </ins> func TestDescribeNode(t *testing.T) { holderIdentity := "holder" <ins> nodeCapacity := mergeResourceLists( getHugePageResourceList("2Mi", "4Gi"), getResourceList("8", "24Gi"), getHugePageResourceList("1Gi", "0"), ) nodeAllocatable := mergeResourceLists( getHugePageResourceList("2Mi", "2Gi"), getResourceList("4", "12Gi"), getHugePageResourceList("1Gi", "0"), ) </ins> fake := fake.NewSimpleClientset( &corev1.Node{

Spec: corev1.NodeSpec{ Unschedulable: true, }, <ins> Status: corev1.NodeStatus{ Capacity:    nodeCapacity, Allocatable: nodeAllocatable, }, </ins> }, &coordinationv1.Lease{ ObjectMeta: metav1.ObjectMeta{

RenewTime:      &metav1.MicroTime{Time: time.Now()}, }, }, <ins> &corev1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name:      "pod-with-resources", Namespace: "foo", }, TypeMeta: metav1.TypeMeta{ Kind: "Pod", }, Spec: corev1.PodSpec{ Containers: []corev1.Container{ { Name:  "cpu-mem", Image: "image:latest", Resources: corev1.ResourceRequirements{ Requests: getResourceList("1", "1Gi"), Limits:   getResourceList("2", "2Gi"), }, }, { Name:  "hugepages", Image: "image:latest", Resources: corev1.ResourceRequirements{ Requests: getHugePageResourceList("2Mi", "512Mi"), Limits:   getHugePageResourceList("2Mi", "512Mi"), }, }, }, }, Status: corev1.PodStatus{ Phase: corev1.PodRunning, }, }, </ins> ) c := &describeClient{T: t, Namespace: "foo", Interface: fake} d := NodeDescriber{c}

t.Errorf("unexpected error: %v", err) } <del> expectedOut := []string{"Unschedulable", "true", "holder"} </del> <ins> expectedOut := []string{"Unschedulable", "true", "holder", `Allocated resources: (Total limits may be over 100 percent, i.e., overcommitted.) Resource           Requests     Limits --------           --------     ------ cpu                1 (25%)      2 (50%) memory             1Gi (8%)     2Gi (16%) ephemeral-storage  0 (0%)       0 (0%) hugepages-1Gi      0 (0%)       0 (0%) hugepages-2Mi      512Mi (25%)  512Mi (25%)`} </ins> for _, expected := range expectedOut { if !strings.Contains(out, expected) { t.Errorf("expected to find %q in output: %q", expected, out)