Adapt code with pytorch.distributed to only one gpu

username: Rubiel1
cooked: <p>Hi,<br>
I am running experiments on other’s people code. They used 16 gpus and the library torch.distributed. I just want to run the code with one gpu, I know it will be slow. Is there a simple way to adapt the code to one GPU without having to learn to use the library pytorch.distributed? at this moment my priority is to see if their code help us, if selected then I’ll focus on that library.</p>
<ul>
<li>Will the library pytorch.distributed automatically detect that I only have one GPU and work on it? No, because it is sending me errors.</li>
</ul>
<blockquote>
<p>Use GPU: 0 for training<br>
Traceback (most recent call last):<br>
File “train.py”, line 97, in <br>
main()<br>
File “train.py”, line 29, in main<br>
mp.spawn(main_worker, nprocs=ngpus_per_node, args=(ngpus_per_node, idx_server, opt))<br>
File “/home/ericd/anaconda/envs/myPytorch/lib/python3.6/site-packages/torch/multiprocessing/spawn.py”, line 171, in spawn<br>
while not spawn_context.join():<br>
File “/home/ericd/anaconda/envs/myPytorch/lib/python3.6/site-packages/torch/multiprocessing/spawn.py”, line 118, in join<br>
raise Exception(msg)<br>
Exception:</p>
<p>– Process 0 terminated with the following error:<br>
Traceback (most recent call last):<br>
File “/home/ericd/anaconda/envs/myPytorch/lib/python3.6/site-packages/torch/multiprocessing/spawn.py”, line 19, in _wrap<br>
fn(i, *args)<br>
File “/home/ericd/tests/CC-FPSE/train.py”, line 37, in main_worker<br>
dist.init_process_group(backend=‘nccl’, init_method=opt.dist_url, world_size=world_size, rank=rank)<br>
File “/home/ericd/anaconda/envs/myPytorch/lib/python3.6/site-packages/torch/distributed/distributed_c10d.py”, line 397, in init_process_group<br>
store, rank, world_size = next(rendezvous_iterator)<br>
File “/home/ericd/anaconda/envs/myPytorch/lib/python3.6/site-packages/torch/distributed/rendezvous.py”, line 120, in _tcp_rendezvous_handler<br>
store = TCPStore(result.hostname, result.port, world_size, start_daemon)<br>
TypeError: <strong>init</strong>(): incompatible constructor arguments. The following argument types are supported:<br>
1. torch.distributed.TCPStore(arg0: str, arg1: int, arg2: int, arg3: bool)</p>
</blockquote>
<p>line 37 is</p>
<p><code>    torch.distributed.init_process_group(backend='nccl', init_method=opt.dist_url, world_size=world_size, rank=rank)</code></p>
<ul>
<li>
<p>How can I adapt the code to my situation? Ideally there is some parameter that I can use and make things compatible.</p>
</li>
<li>
<p>Do I need to find certain lines and modify them? I am afraid that may be the case but I don’t know where or which.</p>
</li>
</ul>
<p>I am working with <a href="https://github.com/xh-liu/CC-FPSE" class="inline-onebox" rel="noopener nofollow ugc">GitHub - xh-liu/CC-FPSE: Code for NeurIPS 2019 paper "Learning to Predict Layout-to-image Conditional Convolutions for Semantic Image Synthesis"</a> but I am not sure if this helps.</p>

username: mrshenli
cooked: <p>It depends on how the code was written. If the model forward function has sth like:</p>
<pre><code class="lang-python">def forward(input):
    x1 = self.layer1(input.to("cuda:0"))
    x2 = self.layer2(input.to("cuda:1"))
    x3 = self.layer3(input.to("cuda:2"))
    return x3
</code></pre>
<p>Then it would certainly fail as it cannot find the device.</p>
<p>If there is nothing like that, you probably can get around by doing sth like:</p>
<pre><code class="lang-auto">torch.distributed.init_process_group(backend='nccl', init_method=opt.dist_url, world_size=1, rank=0)
</code></pre>
<p>You might also need to change <code>opt.dist_url</code> into sth like <code>"tcp://localhost:23456"</code></p>

username: Rubiel1
cooked: <p>It worked, thank you!</p>