How to set backend to 'gloo' on windows

Mo_Balut: <p>I am trying to use two gpus on my windows machine, but I keep getting<br>
<code>raise RuntimeError("Distributed package doesn't have NCCL " "built in") RuntimeError: Distributed package doesn't have NCCL built in</code><br>
I am still new to pytorch and couldnt really find a way of setting the backend to ‘gloo’.  I followed this <a href="https://discuss.huggingface.co/t/run-training-script-in-ddp-using-gloo/15406" rel="noopener nofollow ugc">link</a> by setting the following but still no luck.</p>
<pre><code class="lang-auto">As NLCC is not available on windows I had to tweak the ‘setup_devices’ method of ‘training_args.py’ and write:

torch.distributed.init_process_group(backend=“nccl”) → torch.distributed.init_process_group(backend=“gloo”)

along with the ‘distributed_concat’ in ‘trainer_pt_utils.py’:

dist.all_gather(output_tensors, tensor) → dist.all_gather(output_tensors, tensor if len(tensor.shape) &gt; 0 else tensor[None])
</code></pre>
<p>How to set backend=‘gloo’ and from where?</p>

pbelevich: <p><a class="mention" href="/u/mo_balut">@Mo_Balut</a> could you please show your code?</p>
<p>torch.distributed.init_process_group(backend=“gloo”) is the right way to use gloo</p>

Mo_Balut: <p>thx for replying!</p>
<pre><code class="lang-auto">    os.environ['MASTER_ADDR'] = 'localhost'
    os.environ['MASTER_PORT'] = '12355'
    torch.distributed.init_process_group(backend='gloo',rank=0, world_size=2)
    trainer = pl.Trainer.from_argparse_args(args, checkpoint_callback=checkpoint_callback)

    model = T5Finetuner(hparams=args,
                        train_dataloader=train_dataloader,
                        val_dataloader=val_dataloader,
                        test_dataloader=test_dataloader,
                        orthography=args.orthography)

    trainer.fit(model)
</code></pre>
<p>I am actually not sure what to put on rank. This code doesnt give an error but the command prompt freezes. I am using two gpus. What would be my rank. if I input rank = [0,1]. it gives an error rank should an integer.</p>

pbelevich: <p>If you use PyTorch Lightning you should do the following:</p>
<p><a href="https://pytorch-lightning.readthedocs.io/en/1.4.0/advanced/multi_gpu.html#select-torch-distributed-backend" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch-lightning.readthedocs.io/en/1.4.0/advanced/multi_gpu.html#select-torch-distributed-backend</a></p>

Mo_Balut: <p>Thanks for your fast response.<br>
I didnt get where to add this line  <code>PL_TORCH_DISTRIBUTED_BACKEND=gloo</code></p>

pbelevich: <p>When you run you program in the command line you can prepend it before python train.py:</p>
<pre><code class="lang-bash">pbelevich@pbelevich-mbp ~ % PL_TORCH_DISTRIBUTED_BACKEND=gloo python train.py
</code></pre>

mba: <p>thank you so much! it worked! For some reason the training is slower than one gpu even though two gpus are being used!</p>

polyteddy: <p>I used this in my script to try starting a training session in mmdetection in Windows and received an error:</p>
<p>RuntimeError: trying to initialize the default process group twice!</p>
<p>Would you kindly advise?</p>

Muhammad_Ahtesham_Ul: <p>Please! Can you help me run alpaca model over amazon sage maker?<br>
I can pay you as well</p>