How to decay lr with distributed training using test acc?

MRGAO1996: <p>I use the following code, but I think it may be wrong!<br>
lr_broadcast = torch.tensor([0.0]).cuda()<br>
if distribute:<br>
if local_rank == 0:<br>
lr_broadcast = torch.tensor([optimizer.state_dict()[‘param_groups’][0][‘lr’]]).cuda()<br>
dist.all_reduce(lr_broadcast)<br>
optimizer.state_dict()[‘param_groups’][0][‘lr’] = lr_broadcast.item()<br>
print(local_rank, optimizer.state_dict()[‘param_groups’][0][‘lr’])</p>

mrshenli: <p><a class="mention" href="/u/mrgao1996">@MRGAO1996</a></p>
<p>Could you please format the code with proper indention?</p>
<p>If this line is with in the <code>if local_rank == 0</code> block, then this is indeed wrong. You need to call <code>all_reduce</code> on all processes, as it is a collective communication.</p>
<pre><code class="lang-python">if local_rank == 0:
    dist.all_reduce(lr_broadcast)
</code></pre>

MRGAO1996: <pre><code class="lang-auto">if distribute and local_rank == 0:
    acc = verification(...)
    # lr_sched is torch.optim.lr_scheduler.ReduceLROnPlateau()
    lr_sched.step(acc)
lr_broadcast = torch.tensor([0.0]).cuda()
if distribute:
    if local_rank == 0:
        lr_broadcast = torch.tensor([optimizer.state_dict()['param_groups'][0]['lr']]).cuda()
    dist.all_reduce(lr_broadcast)
    optimizer.state_dict()['param_groups'][0]['lr'] = lr_broadcast.item()
    print(local_rank, optimizer.state_dict()['param_groups'][0]['lr'])
</code></pre>
<p>When I run this code with 2 GPUs, I found two mistakes.<br>
First, dist.all_reduct() failed. I try to print lr_broadcast after all_reduce, but different on GPU0 and GPU1.<br>
Second, after I pass the lr_broadcast value to optimizer.state_dict, I print the lr in optimizer, but the value is differ from lr_broadcast.<br>
So, I got confused.</p>

mrshenli: <p>I noticed the code called <code>.cuda()</code> without specifying a GPU id. Did you set <code>CUDA_VISIBLE_DEVICES</code> or call <code>torch.cuda.set_device()</code>? Could you please share a self-contained repro that shows the error?</p>

MRGAO1996: <p>Well, I knew how to change my code to succeed with “all_reduce”, but I’m still a little comfused.</p>
<p>My code before:(fail)</p>
<pre><code class="lang-auto">from torch.nn.parallel import DistributedDataParallel as DDP

def main():
	processes = []
	for rank in range(world_size):
		p = Process(target=run, args=(rank, world_size))
		p.start()
		processes.append(p)
	for p in processes:
		p.join()

def run(local_rank, world_size):
    os.environ['MASTER_ADDR'] = '127.0.0.1'
    os.environ['MASTER_PORT'] = '29501'
	torch.cuda.set_device(local_rank)
    dist.init_process_group(world_size=world_size, rank=local_rank, backend='nccl')
	net = Net().cuda()
	criterion = ...
	optimizer = ...
	net = DDP(net, device_ids=[local_rank])
	for ep in range(2):
		# train
		net.train()
		for i, (images, labels) in enumerate(train_loader):
			optimizer.zero_grad()
			images = image.cuda()
			labels = labels.cuda()
			outputs = net(images)
			loss = criterion(outputs, labels)
			loss.backward()
			optimizer.step()
		# verification
		if local_rank != 0:
            val_acc = torch.tensor([0.0]).cuda()
		# only do verification on rank0
		if local_rank == 0:
			net.eval()
			# simplify verification simulation
			input = torch.randn(8, 3, 112, 112).cuda()
			with torch.no_grad():
				output = net(input)
			val_acc = torch.tensor([0.0009]).cuda()
		dist.all_reduct(val_acc)
		print(local_rank, val_acc)

main()
</code></pre>
<p>The code below shows:</p>
<pre><code class="lang-auto">1 tensor([0.0], device='cuda:1')
0 tensor([0.0009], device='cuda:0')
</code></pre>
<p>And, in epoch 2, local_rank 1 will hang up.</p>
<p>My solution is that, change the line “output = net(input)” to “output = net.module(input)”, it succeed.</p>
<pre><code class="lang-auto">0 tensor([0.0009], device='cuda:0')
1 tensor([0.0009], device='cuda:1')
</code></pre>
<p>So I want to know why this happen?</p>

MRGAO1996: <p>Could you please explain this <img src="https://discuss.pytorch.org/images/emoji/apple/slightly_smiling_face.png?v=9" title=":slightly_smiling_face:" class="emoji" alt=":slightly_smiling_face:"></p>

mrshenli: <p>Hey <a class="mention" href="/u/mrgao1996">@MRGAO1996</a></p>
<p>Does your model use any buffers (e.g., running mean in <code>BatchNorm</code>)?</p>
<p>As you have wrapped the second forward with <code>with torch.no_grad():</code>, rank 0 will skip the code in <code>if torch.is_grad_enabled() and ...</code> branch (see DDP forward below), but the behavior of the next <code>forward</code> will be different on two ranks, as rank 0 would skip <code>_sync_params()</code> but rank 1 would execute that. But this method should only make a difference when your model has buffers.</p>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/blob/d5ae4a07ef5b2e77cf51737fb0a3aafc2e71231d/torch/nn/parallel/distributed.py#L562-L590" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/d5ae4a07ef5b2e77cf51737fb0a3aafc2e71231d/torch/nn/parallel/distributed.py#L562-L590" target="_blank" rel="nofollow noopener">pytorch/pytorch/blob/d5ae4a07ef5b2e77cf51737fb0a3aafc2e71231d/torch/nn/parallel/distributed.py#L562-L590</a></h4>
<pre class="onebox"><code class="lang-py"><ol class="start lines" start="562" style="counter-reset: li-counter 561 ;">
<li>def forward(self, *inputs, **kwargs):</li>
<li>    if self.require_forward_param_sync:</li>
<li>        self._sync_params()</li>
<li>
</li><li>    if self.device_ids:</li>
<li>        inputs, kwargs = self.scatter(inputs, kwargs, self.device_ids)</li>
<li>        if len(self.device_ids) == 1:</li>
<li>            output = self.module(*inputs[0], **kwargs[0])</li>
<li>        else:</li>
<li>            outputs = self.parallel_apply(self._module_copies[:len(inputs)], inputs, kwargs)</li>
<li>            output = self.gather(outputs, self.output_device)</li>
<li>    else:</li>
<li>        output = self.module(*inputs, **kwargs)</li>
<li>
</li><li>    if torch.is_grad_enabled() and self.require_backward_grad_sync:</li>
<li>        self.require_forward_param_sync = True</li>
<li>        # We'll return the output object verbatim since it is a freeform</li>
<li>        # object. We need to find any tensors in this object, though,</li>
<li>        # because we need to figure out which parameters were used during</li>
<li>        # this forward pass, to ensure we short circuit reduction for any</li>
</ol></code></pre>

  This file has been truncated. <a href="https://github.com/pytorch/pytorch/blob/d5ae4a07ef5b2e77cf51737fb0a3aafc2e71231d/torch/nn/parallel/distributed.py#L562-L590" target="_blank" rel="nofollow noopener">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>