Change:
<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: Which test(s) are failing: Since when has it been failing: Testgrid link: Reason for failure: Anything else we need to know: Can't find this in Triage Example spyglass links: /sig scheduling /cc /priority important-soon
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I don't quite think this test should be owned by SIG-Scheduling. Most of the failures happen in the Nvidia GPU setup phase. BTW: although irrelevant, last time we see a failure of this test is .
Searching keyword "UnexpectedAdmissionError" gets the following occurrences:
/sig node cc in case you're aware of any recent changes that may have caused this failure.
From the I can see a bunch of: Checking the kubelet log, I can see the gpu capacity has been changing And before the capacity changed to 0, we can see the kubelet was attempting to allocate 1 GPU to the test pod, but some NPE was thrown, and kubelet stopped/restarted.
The nil pointer is from here: , which is a new change introduced by
The current device plugin (the one defined ) is using an old version of device plugin api (see ), which does not have field under . So calling will end up with NPE.
Thanks , the analysis makes great sense to me. Could you please work on a fix to address this? (Probably updating )
Hi , I am still looking at the code, because the code does have the logic to handle the case where device plugin does not return affinity: Also, even if the Topology is not defined, I am not sure will throw NPE () (please correct me if I am wrong here). I will need to look more closely at the code to find out where the NPE is from. Updating the might take long time because we will need some testing to make sure it doesn't break anything. Currently I am thinking of identifying the source of NPE, and make sure can handle that case.
/cc
I submitted a fix. My fault, it was a real case when pointer to Topology could be a nil. But still fails: I don't observe SIGSEGV now in , but 3 problems there: 1113 09:26:19. 1336 ] failed to run Kubelet: cannot create certificate signing request: is forbidden: User "kubelet" cannot create resource "certificatesigningrequests" in API group "" at the cluster scope of nvidia-driver-installer stops and restarts after a while. 's () NoopManager is using
can you show full error msg of "NVIDIA GPU metrics will not be available: %s"?
I1113 15:43:26. 1338 ] Found device with vendorID "0x10de" W1113 15:43:26. 1338 ] NVIDIA GPU metrics will not be available: Could not initialize NVML: could not load NVML library I1113 15:43:26. 1338 ] Machine: {Timestamp:2020-11-13 15:43:26. +0000 UTC m=+5. NumCores:2 NumPhysicalCores:1 NumSockets:1 CpuFrequency: MemoryCapacity: MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:2048 NumPages:0}] MachineID: SystemUUID:-9824-3a64-f41f- BootID:-f891-4c0b-a34d- Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:39 Capacity: Type:vfs Inodes: HasInodes:true} {Device:/dev/sda8 DeviceMajor:8 DeviceMinor:8 Capacity: Type:vfs Inodes:4096 HasInodes:true} {Device:/dev/sda1 DeviceMajor:8 DeviceMinor:1 Capacity: Type:vfs Inodes: HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:19 Capacity: Type:vfs Inodes: HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:21 Capacity: Type:vfs Inodes: HasInodes:true} {Device:/sys/fs/cgroup DeviceMajor:0 DeviceMinor:22 Capacity: Type:vfs Inodes: HasInodes:true} {Device:/etc/machine-id DeviceMajor:0 DeviceMinor:21 Capacity: Type:vfs Inodes: HasInodes:true} {Device:/mnt/disks DeviceMajor:0 DeviceMinor:38 Capacity: Type:vfs Inodes: HasInodes:true} {Device:overlayfs0-43 DeviceMajor:0 DeviceMinor:43 Capacity: Type:vfs Inodes: HasInodes:true} {Device:/var/lib/cloud DeviceMajor:0 DeviceMinor:45 Capacity: Type:vfs Inodes: HasInodes:true} {Device:/dev/root DeviceMajor:253 DeviceMinor:0 Capacity: Type:vfs Inodes: HasInodes:true}] DiskMap:map[253:0:{Name:dm-0 Major:253 Minor:0 Size: Scheduler:none} 8:0:{Name:sda Major:8 Minor:0 Size: Scheduler:mq-deadline} 9:0:{Name:md0 Major:9 Minor:0 Size:0 Scheduler:none}] NetworkDevices:[{Name:eth0 MacAddress:42:01:0a:28:00:03 Speed:-1 Mtu:1460}] Topology:[{Id:0 Memory: HugePages:[{PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 1] Caches:[{Size: Type:Data Level:1} {Size: Type:Instruction Level:1} {Size: Type:Unified Level:2}] SocketID:0}] Caches:[{Size: Type:Unified Level:3}]}] CloudProvider:GCE InstanceType:n1-standard-2 InstanceID:} I1113 15:43:26. 1338 ] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. I1113 15:43:26. 1338 ] Version: {KernelVersion:5.4.49+ ContainerOsVersion:Container-Optimized OS from Google DockerVersion:Unknown DockerAPIVersion:Unknown CadvisorVersion: CadvisorRevision:} I guess kubelet can't load in cAdvisor the NVML library.
I have looked into this issue actually. This is caused by a recent change on cadvisor: We used to return a real nvidiaManager but now only a Noop one if the NVML lib is not there. And currently the NVML lib is loaded by daemonset, which will not there when the kubelet starts. This PR should fix it: But btw, this bug will not fail the test because it will only cause missing GPU usage metrics. The plan is to cherry-pick the fix above to cadvisor 0.37, and then update the vendor for kubernetes

Discussion:
// Update internal cached podDevices state. m.mutex.Lock() for dev := range allocDevices { <ins> if m.allDevices[resource][dev].Topology == nil || len(m.allDevices[resource][dev].Topology.Nodes) == 0 { allocDevicesWithNUMA[0] = append(allocDevicesWithNUMA[0], dev) continue } </ins> for idx := range m.allDevices[resource][dev].Topology.Nodes { node := m.allDevices[resource][dev].Topology.Nodes[idx] allocDevicesWithNUMA[node.ID] = append(allocDevicesWithNUMA[node.ID], dev)

opts: nil, } } <del> testManager.allDevices[res.resourceName] = makeDevice(res.devs) </del> <ins> testManager.allDevices[res.resourceName] = makeDevice(res.devs, res.topology) </ins> } return testManager, nil

resourceName     string resourceQuantity resource.Quantity devs             checkpoint.DevicesPerNUMA <ins> topology         bool </ins> } func TestPodContainerDeviceAllocation(t *testing.T) {

resourceName:     "domain1.com/resource1", resourceQuantity: *resource.NewQuantity(int64(2), resource.DecimalSI), devs:             checkpoint.DevicesPerNUMA{0: []string{"dev1", "dev2"}}, <ins> topology:         true, </ins> } res2 := TestResource{ resourceName:     "domain2.com/resource2", resourceQuantity: *resource.NewQuantity(int64(1), resource.DecimalSI), devs:             checkpoint.DevicesPerNUMA{0: []string{"dev3", "dev4"}}, <ins> topology:         false, </ins> } testResources := make([]TestResource, 2) testResources = append(testResources, res1)

resourceName:     "domain1.com/resource1", resourceQuantity: *resource.NewQuantity(int64(2), resource.DecimalSI), devs:             checkpoint.DevicesPerNUMA{0: []string{"dev1", "dev2"}}, <ins> topology:         false, </ins> } res2 := TestResource{ resourceName:     "domain2.com/resource2", resourceQuantity: *resource.NewQuantity(int64(1), resource.DecimalSI), devs:             checkpoint.DevicesPerNUMA{0: []string{"dev3", "dev4"}}, <ins> topology:         true, </ins> } testResources := make([]TestResource, 2) testResources = append(testResources, res1)

resourceName:     "domain1.com/resource1", resourceQuantity: *resource.NewQuantity(int64(2), resource.DecimalSI), devs:             checkpoint.DevicesPerNUMA{0: []string{"dev1", "dev2"}}, <ins> topology:         false, </ins> } as := require.New(t) podsStub := activePodsStub{

} } <del> func makeDevice(devOnNUMA checkpoint.DevicesPerNUMA) map[string]pluginapi.Device { </del> <ins> func makeDevice(devOnNUMA checkpoint.DevicesPerNUMA, topology bool) map[string]pluginapi.Device { </ins> res := make(map[string]pluginapi.Device) <ins> var topologyInfo *pluginapi.TopologyInfo </ins> for node, devs := range devOnNUMA { <ins> if topology { topologyInfo = &pluginapi.TopologyInfo{Nodes: []*pluginapi.NUMANode{{ID: node}}} } else { topologyInfo = nil } </ins> for idx := range devs { <del> res[devs[idx]] = pluginapi.Device{ID: devs[idx], Topology: &pluginapi.TopologyInfo{Nodes: []*pluginapi.NUMANode{{ID: node}}}} </del> <ins> res[devs[idx]] = pluginapi.Device{ID: devs[idx], Topology: topologyInfo} </ins> } } return res