Code change:
I am trying out For that I enabled kubelet memory manager, cpu manager and topology manager. This is tried on a dual socket NUMA server KUBELETKUBEADMARGS="--container-runtime=remote --container-runtime- --cpu-manager-policy=static --feature-gates=RotateKubeletServerCertificate=true,AllAlpha=false,RemoveSelfLink=false,EphemeralContainers=true,BoundServiceAccountTokenVolume=false,KubeletPodResourcesGetAllocatable=true,TopologyManager=true,MemoryManager=true --kube-api-burst=40 --kube-api-qps=20 --kube-reserved=cpu=4,memory=4Gi --memory-manager-policy=static --reserved-cpus=0,1,2,3 --reserved-memory 0:memory=4Gi,1:memory=4196Mi --system-reserved=cpu=4,memory=4Gi --topology-manager-policy=single-numa-node" topologyManagerPolicy: single-numa-node is set on This throws an error 2021-10-20T11:28:49.+00:00 localhost kubelet[]: E1020 11:28:49. ] "Failed to parse kubelet flag" err="invalid argument "0:memory=4Gi,1:memory=4196Mi" for "--reserved-memory" flag: the reserved memory has incorrect format, expected numaNodeID:type=quantity[,type=quantity...], got 0:memory=4Gi,1:memory=4196Mi" This is on kubernetes 1.21.1 Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2021-09-29T11:36:35Z", GoVersion:"go1.16.4", Compiler:"gc", Platform:"linux/amd64"} Any hints on specifying "reserved-memory" for multi-socket NUMA system ???? One way is specifying multiple "reserved-memory" like But problem with this is is map, so it will get over-written. we should be able to specify reserved memory in a multi socket system. Enable above flags and run kubelet No response Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.1", GitCommit:"", GitTreeState:"clean", BuildDate:"2021-09-29T11:36:35Z", GoVersion:"go1.16.4", Compiler:"gc", Platform:"linux/amd64"} none SLES 15 sp2 <details</detailscontainerd No response
/sig node
this is the right way to config
Problem with that is KubeletExtraArgs map[string]string is a map, so multiple "reserved-memory" will get over-written.
there is a tracking issue for kubeadm's api to support multiple arguments with the same name: but it is a backwards incompatible change. you should beusing the KubeletConfiguration for this instead:
/remove-kind support /kind bug /assign Looks like a bug in a parsing logic + better error would be beneficial
Is this a parsing bug?, if yes can you just point me where I need to fix this. I will try to fix this in the downstream release(our internal release).
Here the parsing logic - and here it is failing - Give me know if you need some additional help.
Are we planning to add support for multiple numa nodes in single "--reserved-memory" flag, or just displaying a better error msg?
We should support multiple NUMA nodes under the same --reserved-memory flag.
/triage accepted
Can you provide a sample configuration for 2 numa nodes? "." comma seperator is already there for single numa configuration.
Yes, it is the problem, initially we needed to split it by comma, but probably it will not work as expected, so maybe we can split it by and update documentation. Something like:
can you please review the PR and provide your comments

Related discussion:
text: return nil } <del> numaNodeReservation := strings.Split(s, ":") if len(numaNodeReservation) != 2 { return fmt.Errorf("the reserved memory has incorrect format, expected numaNodeID:type=quantity[,type=quantity...], got %s", s) } memoryTypeReservations := strings.Split(numaNodeReservation[1], ",") if len(memoryTypeReservations) < 1 { return fmt.Errorf("the reserved memory has incorrect format, expected numaNodeID:type=quantity[,type=quantity...], got %s", s) } numaNodeID, err := strconv.Atoi(numaNodeReservation[0]) if err != nil { return fmt.Errorf("failed to convert the NUMA node ID, exptected integer, got %s", numaNodeReservation[0]) } memoryReservation := kubeletconfig.MemoryReservation{ NumaNode: int32(numaNodeID), Limits:   map[v1.ResourceName]resource.Quantity{}, } for _, reservation := range memoryTypeReservations { limit := strings.Split(reservation, "=") if len(limit) != 2 { return fmt.Errorf("the reserved limit has incorrect value, expected type=quantatity, got %s", reservation) </del> <ins> numaNodeReservations := strings.Split(s, ";") for _, reservation := range numaNodeReservations { numaNodeReservation := strings.Split(reservation, ":") if len(numaNodeReservation) != 2 { return fmt.Errorf("the reserved memory has incorrect format, expected numaNodeID:type=quantity[,type=quantity...], got %s", reservation) </ins> } <del> resourceName := v1.ResourceName(limit[0]) if resourceName != v1.ResourceMemory && !corev1helper.IsHugePageResourceName(resourceName) { return fmt.Errorf("memory type conversion error, unknown type: %q", resourceName) </del> <ins> memoryTypeReservations := strings.Split(numaNodeReservation[1], ",") if len(memoryTypeReservations) < 1 { return fmt.Errorf("the reserved memory has incorrect format, expected numaNodeID:type=quantity[,type=quantity...], got %s", reservation) </ins> } <del> q, err := resource.ParseQuantity(limit[1]) </del> <ins> numaNodeID, err := strconv.Atoi(numaNodeReservation[0]) </ins> if err != nil { <del> return fmt.Errorf("failed to parse the quantatity, expected quantatity, got %s", limit[1]) </del> <ins> return fmt.Errorf("failed to convert the NUMA node ID, exptected integer, got %s", numaNodeReservation[0]) </ins> } <del> memoryReservation.Limits[v1.ResourceName(limit[0])] = q } </del> <ins> memoryReservation := kubeletconfig.MemoryReservation{ NumaNode: int32(numaNodeID), Limits:   map[v1.ResourceName]resource.Quantity{}, } </ins> <del> *v.Value = append(*v.Value, memoryReservation) </del> <ins> for _, memoryTypeReservation := range memoryTypeReservations { limit := strings.Split(memoryTypeReservation, "=") if len(limit) != 2 { return fmt.Errorf("the reserved limit has incorrect value, expected type=quantatity, got %s", memoryTypeReservation) } resourceName := v1.ResourceName(limit[0]) if resourceName != v1.ResourceMemory && !corev1helper.IsHugePageResourceName(resourceName) { return fmt.Errorf("memory type conversion error, unknown type: %q", resourceName) } q, err := resource.ParseQuantity(limit[1]) if err != nil { return fmt.Errorf("failed to parse the quantatity, expected quantatity, got %s", limit[1]) } memoryReservation.Limits[v1.ResourceName(limit[0])] = q } *v.Value = append(*v.Value, memoryReservation) } </ins> return nil }