SchedMD/slurm — Issue #11: "File" entry ignored in gres.conf
Author: nbigaouette
Created: 2012-01-23 21:42:46+00:00
State: open

The "File" entry in /etc/slurm/gres.conf seems to be ignored when parsed.

I have three video cards in my system of which two should be controlled by slurm. The third video card drives the display and is underpowered. So I configured /etc/slurm/gres.conf like this:
Name=gpu File=/dev/nvidia0
Name=gpu File=/dev/nvidia2
Note that /dev/nvidia1 is my display card.

When launched, slurmd checks if the files exist.

But when a job a launched, the "File" entry is ignored. The gpus are instead allocated incrementally. So a job asking for 2 gpus will run on /dev/nvidia0 and /dev/nvidia1.

If I set gres.conf to contain this instead:
Name=gpu File=/dev/nvidia0
Name=gpu File=/dev/nvidia2
Name=gpu File=/dev/nvidia1
and submit a job asking for 3 gpus, is set to "CUDA_VISIBLE_DEVICES: 0,1,2" and not "CUDA_VISIBLE_DEVICES: 0,2,1".

In the documentation, it is said that the files should be in increasing order, but this is actually ignored...

Until this is fixed, I cannot use half of the machine here...