akankshakshurai/cuda-error-resolution-analysis huggingface dataset:
Can you perform testing while the model is in DistributedDataParallel mode?

txcode: <p>I wanted to ask, does utilizing DistributedDataParallel impact validation/testing? I’m wondering if it is still possible to “pause” after each epoch and test the model.</p>
<p>Does DistributedDataParallel prevent testing the model during the training loop (after an epoch)?</p>
<p>Since the data and model are trained on different devices, I was unsure if there was an issue with testing the model.</p>
<p>Also, if you can test, should it be done on only a single node, or how should the testing loop be incorporated? For all training agents?</p>
<p>For reference the docs are shown <a href="https://pytorch.org/tutorials/intermediate/ddp_tutorial.html" rel="noopener nofollow ugc">here</a>.</p>

mrshenli: <aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/pytorch/blob/d9106116aa5e399f7d63feeb7fc77f92a076dd93/torch/nn/parallel/distributed.py#L939-L949">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/blob/d9106116aa5e399f7d63feeb7fc77f92a076dd93/torch/nn/parallel/distributed.py#L939-L949" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/d9106116aa5e399f7d63feeb7fc77f92a076dd93/torch/nn/parallel/distributed.py#L939-L949" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/d9106116aa5e399f7d63feeb7fc77f92a076dd93/torch/nn/parallel/distributed.py#L939-L949</a></h4>



  <pre class="onebox">    <code class="lang-py">
      <ol class="start lines" start="939" style="counter-reset: li-counter 938 ;">
          <li>self.require_forward_param_sync = True</li>
          <li># We'll return the output object verbatim since it is a freeform</li>
          <li># object. We need to find any tensors in this object, though,</li>
          <li># because we need to figure out which parameters were used during</li>
          <li># this forward pass, to ensure we short circuit reduction for any</li>
          <li># unused parameters. Only if `find_unused_parameters` is set.</li>
          <li>if self.find_unused_parameters and not self.static_graph:</li>
          <li>    # Do not need to populate this for static graph.</li>
          <li>    self.reducer.prepare_for_backward(list(_find_tensors(output)))</li>
          <li>else:</li>
          <li>    self.reducer.prepare_for_backward([])</li>
      </ol>
    </code>
  </pre>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>The code above would make <code>DistributedDataParallel</code> expecting a backward after the current forward. But you can run the validation/testing forward within a <code>with torch.no_grad():</code> context to force into a different branch.</p>
Torch.distributed.broadcast failed when broadcast in multi GPUs in one machine

Bone: <p>I try to broadcast a tensor(which is a scalar in practice) from rank0 to other ranks in a machine. But i found the tensor in rank1,2,3 not equals to rank0.</p>
<h3>
<a name="code-1" class="anchor" href="#code-1"></a>Code</h3>
<pre><code class="lang-auto">    if ddp.get_rank() == 0:
        model.eval()
        fit_accurary = utils.AverageMeter()
        for batch_idx, (inputs, targets) in enumerate(testLoader):
            inputs, targets = inputs.to(device), targets.to(device)
            with torch.no_grad():
                outputs = model(inputs)
            predicted = utils.accuracy(outputs, targets,topk=(1,5))
            fit_accurary.update(predicted[1], inputs.size(0))

        if fit_accurary.avg == 0:
            fit_accurary.avg = 0.01

        if fit_accurary.avg &gt; best_honey.fitness:
            best_honey_state = copy.deepcopy(model.module.state_dict())
            best_honey.code = copy.deepcopy(honey)
            best_honey.fitness = fit_accurary.avg
        avg_acc = float(fit_accurary.avg)
    else:
        avg_acc = float(0)
    avg_acc = (torch.tensor(avg_acc, dtype=torch.float)).to(device)
    dist.broadcast(avg_acc, src=0)
    #avg_acc = utils.send_and_wait(r=0, data=[avg_acc])
    avg_acc = avg_acc.cpu().detach().numpy().item()
    print(avg_acc)
    return avg_acc
</code></pre>
<h3>
<a name="results-2" class="anchor" href="#results-2"></a>Results</h3>
<p>0.009999999776482582 rank0<br>
-0.0011623682221397758 rank1<br>
-0.0011623682221397758 rank2<br>
-0.0011623682221397758 rank3</p>

mrshenli: <aside class="quote no-group" data-username="Bone" data-post="1" data-topic="140059">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/b/65b543/48.png" class="avatar"> Bone:</div>
<blockquote>
<p>0.009999999776482582 rank0<br>
-0.0011623682221397758 rank1<br>
-0.0011623682221397758 rank2<br>
-0.0011623682221397758 rank3</p>
</blockquote>
</aside>
<p>Unless <code>avg_acc</code> are already synced on rank1-3, otherwise it looks like rank0 has successfully broadcast to the other three ranks (so that they share the same value) and then the <code>avg_acc</code> was somehow modified afterwards before prints.</p>
<p>Can you try add a stream sync after broadcast and print immediately after that? Sth like:</p>
<pre data-code-wrap="python"><code class="lang-python">dist.broadcast(avg_acc, src=0)
torch.cuda.current_stream(device).synchronize()
print(avg_acc)
</code></pre>
<p>BTW, would I be correct that each process/rank exclusively occupies one device?</p>
<pre><code class="lang-auto"></code></pre>
Type torch::jit::Module converted to torch._C.ScriptModule

EFans: <p>Hi,<br>
I’m trying to develop some torch.jit modules with c++，and then use them in python with pybind.<br>
I found that the orginal method of torch._C.ScriptModule._create_method_from_trace<br>
was created with arg types torch::jit::Module in c++.  But the corresponding method arg type was torch._c.ScriptModule in python.<br>
How dose this conversion happens?<br>
I try to define a method in c++ with arg type torch::jit::Module. When exported to python, it’s arg type still torch::jit::Module.<br>
What’s missed when building the program?</p>

EFans: <p>Build with these cxx flags can fix:</p>
<hr>
<pre><code> cmake_cxx_flags: List[str] = []
    for name in ["COMPILER_TYPE", "STDLIB", "BUILD_ABI"]:
        val = getattr(torch._C, f"_PYBIND11_{name}")
        if val is not None:
            cmake_cxx_flags += [f'-DPYBIND11_{name}=\\"{val}\\"']
        print("val: ", val)
        print("cmake_cxx_flags: ", cmake_cxx_flags)</code></pre>