DDP with slurm CUDA error: invalid device ordinal

username: shrutishrestha
cooked: <p>Hi, I am doing DistributedDataParallel with slurm. My code is written below. The error I am getting is RuntimeError: CUDA error: invalid device ordinal at line 14.</p>
<ol>
<li>import os</li>
<li>import hostlist</li>
</ol>
<h1><a name="get-slurm-variables-1" class="anchor" href="#get-slurm-variables-1"></a>get SLURM variables</h1>
<ol start="3">
<li>rank = int(os.environ[‘SLURM_PROCID’])</li>
<li>local_rank = int(os.environ[‘SLURM_LOCALID’])</li>
<li>size = int(os.environ[‘SLURM_NTASKS’])</li>
<li>cpus_per_task = int(os.environ[‘SLURM_CPUS_PER_TASK’])</li>
</ol>
<h1><a name="get-node-list-from-slurm-2" class="anchor" href="#get-node-list-from-slurm-2"></a>get node list from slurm</h1>
<ol start="7">
<li>hostnames = hostlist.expand_hostlist(os.environ[‘SLURM_JOB_NODELIST’])</li>
</ol>
<h1><a name="get-ids-of-reserved-gpu-3" class="anchor" href="#get-ids-of-reserved-gpu-3"></a>get IDs of reserved GPU</h1>
<ol start="8">
<li>gpu_ids = os.environ[‘SLURM_STEP_GPUS’].split(“,”)</li>
</ol>
<h1><a name="define-master_add-master_port-4" class="anchor" href="#define-master_add-master_port-4"></a>define MASTER_ADD &amp; MASTER_PORT</h1>
<ol start="9">
<li>
<p>os.environ[‘MASTER_ADDR’] = hostnames[0]</p>
</li>
<li>
<p>os.environ[‘MASTER_PORT’] = str(12345 + int(min(gpu_ids))) # to avoid port conflict on the same node</p>
</li>
<li>
<p>NODE_ID = os.environ[‘SLURM_NODEID’]</p>
</li>
<li>
<p>MASTER_ADDR = os.environ[‘MASTER_ADDR’]</p>
</li>
<li>
<p>dist.init_process_group(“nccl”, init_method=‘env://’, world_size=size, rank=rank)</p>
</li>
<li>
<p>torch.cuda.set_device(local_rank)</p>
</li>
</ol>
<p>I am using this below configuration in slurm<br>
15. <span class="hashtag-raw">#SBATCH</span> --nodes=2<br>
16. <span class="hashtag-raw">#SBATCH</span> --ntasks-per-node=4<br>
17. <span class="hashtag-raw">#SBATCH</span> --gres=gpu:2<br>
18. <span class="hashtag-raw">#SBATCH</span> --cpus-per-task=4<br>
19. <span class="hashtag-raw">#SBATCH</span> --hint=nomultithread2<br>
20. <span class="hashtag-raw">#SBATCH</span> --gres=gpu:V100:2</p>
<h1><a name="i-get-the-outputs-as-5" class="anchor" href="#i-get-the-outputs-as-5"></a>I get the outputs as</h1>
<p>rank 0</p>
<p>local_rank 0</p>
<p>size 8</p>
<p>cpus_per_task 4</p>
<p>gpu_ids [‘2’, ‘3’]</p>
<p>hostnames [‘acidsgcn001’, ‘acidsgcn002’]</p>
<blockquote>
<blockquote>
<blockquote>
<p>Training on 2 nodes and 8 processes, master node is acidsgcn001</p>
</blockquote>
</blockquote>
</blockquote>
<ul>
<li>Process 0 corresponds to GPU 0 of node 0</li>
</ul>
<p>rank 3</p>
<p>local_rank 3</p>
<p>size 8</p>
<p>cpus_per_task 4</p>
<p>gpu_ids [‘2’, ‘3’]</p>
<p>hostnames [‘acidsgcn001’, ‘acidsgcn002’]</p>
<ul>
<li>Process 3 corresponds to GPU 3 of node 0</li>
</ul>
<p>rank 2</p>
<p>local_rank 2</p>
<p>size 8</p>
<p>cpus_per_task 4</p>
<p>gpu_ids [‘2’, ‘3’]</p>
<p>hostnames [‘acidsgcn001’, ‘acidsgcn002’]</p>
<ul>
<li>Process 2 corresponds to GPU 2 of node 0</li>
</ul>
<p>rank 1</p>
<p>local_rank 1</p>
<p>size 8</p>
<p>cpus_per_task 4</p>
<p>gpu_ids [‘2’, ‘3’]</p>
<p>hostnames [‘acidsgcn001’, ‘acidsgcn002’]</p>
<ul>
<li>Process 1 corresponds to GPU 1 of node 0</li>
</ul>
<p>rank 4</p>
<p>local_rank 0</p>
<p>size 8</p>
<p>cpus_per_task 4</p>
<p>gpu_ids [‘2’, ‘3’]</p>
<p>hostnames [‘acidsgcn001’, ‘acidsgcn002’]</p>
<ul>
<li>Process 4 corresponds to GPU 0 of node 1</li>
</ul>
<p>rank 6</p>
<p>local_rank 2</p>
<p>size 8</p>
<p>cpus_per_task 4</p>
<p>gpu_ids [‘2’, ‘3’]</p>
<p>hostnames [‘acidsgcn001’, ‘acidsgcn002’]</p>
<ul>
<li>Process 6 corresponds to GPU 2 of node 1</li>
</ul>
<p>rank 7</p>
<p>local_rank 3</p>
<p>size 8</p>
<p>cpus_per_task 4</p>
<p>gpu_ids [‘2’, ‘3’]</p>
<p>hostnames [‘acidsgcn001’, ‘acidsgcn002’]</p>
<ul>
<li>Process 7 corresponds to GPU 3 of node 1</li>
</ul>
<p>rank 5</p>
<p>local_rank 1</p>
<p>size 8</p>
<p>cpus_per_task 4</p>
<p>gpu_ids [‘2’, ‘3’]</p>
<p>hostnames [‘acidsgcn001’, ‘acidsgcn002’]</p>
<ul>
<li>Process 5 corresponds to GPU 1 of node 1</li>
</ul>