How multi-threading is managed within ddp?

username: sbelharbi
cooked: <p>hi,<br>
i have a c++ loss-wrapped in python.<br>
here is some stats:<br>
in all these cases, <a href="https://pytorch.org/docs/stable/generated/torch.nn.parallel.DistributedDataParallel.html#torch.nn.parallel.DistributedDataParallel" rel="noopener nofollow ugc">ddp</a> is used. but we can choose to use one or two gpus.</p>
<p>here we show the forward time in the loss. more specifically, part of the code in the forward.<br>
that part operates on cpu. so, gpu is not involved since we convert the output gpu tensor from previous computation to cpu().numpy(). then, computations are carried on cpu.<br>
time is measured using</p>
<pre><code class="lang-python">
def forward(x):
    torch.cuda.synchronize()
    start_event = torch.cuda.Event(enable_timing=True) 
    end_event = torch.cuda.Event(enable_timing=True)
    start_event.record()
    # cpu region ---
    compute_on_cpu()
    # cpu region ---
    end_event.record()
    torch.cuda.synchronize()
    elapsed_time_ms = start_event.elapsed_time(end_event)
    print('time cpu: {}'.format(elapsed_time_ms))
</code></pre>
<p>1 gpu:</p>
<ul>
<li>multi-threading is on: 70ms</li>
<li>multi-threading is off: 500ms.</li>
</ul>
<p>2gpus:</p>
<ul>
<li>multi-threading is on: 500ms</li>
<li>multi-threading is off: 500ms.</li>
</ul>
<p>the loss uses the maximum number of threads (<a href="https://www.openmp.org/spec-html/5.0/openmpsu112.html" rel="noopener nofollow ugc">openmp</a> <code>omp_get_max_threads()</code>) which is 48 in this case.</p>
<p>it looks like when using multi-processes (multi-gpus), multi-threading is not working…<br>
any idea why (gil bottleneck???)? and how to fix this?<br>
the whole point of using ddp is to speedup computations.</p>
<p>thanks</p>

username: Yanli_Zhao
cooked: <p>openmp threads are mostly for cpu computation, for GPU computation, you need to make sure tensors and computation are on GPUs, it will launch CUDA kernels and compute in parallel.</p>

username: sbelharbi
cooked: <p>yes,<br>
but this still does not answer the question to why c++ multi-threading seems to be turned off when using ddp + multigpus.<br>
i dont not whether the threads of each process block each other or something else.</p>
<p>the rest of computation is done on gpu where all tensors live.<br>
only <code>compute_cpu()</code> is done exclusively on cpu because it is simply a c++ cpu implementation. dont have yet a gpu implementation.</p>

username: sbelharbi
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> any idea why this is happening?<br>
this is the reason i was looking for a cuda extension in the <a href="https://discuss.pytorch.org/t/c-cuda-custom-function-runtimeerror-cuda-error-invalid-device-function/131085">other</a> thread.</p>
<p>this is a cpu c++ code, and has nothing to do with the other thread.</p>
<p>basically, the function <code>compute_on_cpu()</code> calls the c++ function:</p>
<aside class="onebox githubblob" data-onebox-src="https://github.com/meng-tang/rloss/blob/1caa759e568db2c7209ab73e73ac039ea3d7101c/pytorch/wrapper/bilateralfilter/bilateralfilter.cpp#L42">
  <header class="source">

      <a href="https://github.com/meng-tang/rloss/blob/1caa759e568db2c7209ab73e73ac039ea3d7101c/pytorch/wrapper/bilateralfilter/bilateralfilter.cpp#L42" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/meng-tang/rloss/blob/1caa759e568db2c7209ab73e73ac039ea3d7101c/pytorch/wrapper/bilateralfilter/bilateralfilter.cpp#L42" target="_blank" rel="noopener nofollow ugc">meng-tang/rloss/blob/1caa759e568db2c7209ab73e73ac039ea3d7101c/pytorch/wrapper/bilateralfilter/bilateralfilter.cpp#L42</a></h4>



  <pre class="onebox">    <code class="lang-cpp">
      <ol class="start lines" start="32" style="counter-reset: li-counter 31 ;">
          <li>        for(int i=0;i&lt;W*H;i++)</li>
          <li>            in_p[i] = in[i+k*W*H];</li>
          <li>        lattice.compute(out_p, in_p, 1);</li>
          <li>        for(int i=0;i&lt;W*H;i++)</li>
          <li>            out[i+k*W*H] = out_p[i];</li>
          <li>    }</li>
          <li>    delete [] out_p;</li>
          <li>    delete [] in_p;</li>
          <li>}</li>
          <li>
          </li>
<li class="selected">void bilateralfilter_batch(float * images, int len_images, float * ins, int len_ins, float * outs, int len_outs,</li>
          <li>                              int N, int K, int H, int W, float sigmargb, float sigmaxy){</li>
          <li>    </li>
          <li>    const int maxNumThreads = omp_get_max_threads();</li>
          <li>    //printf("Maximum number of threads for this machine: %i\n", maxNumThreads);</li>
          <li>    omp_set_num_threads(std::min(maxNumThreads,N));</li>
          <li>    </li>
          <li>    #pragma omp parallel for</li>
          <li>    for(int n=0;n&lt;N;n++){</li>
          <li>        bilateralfilter(images+n*3*H*W, 3*H*W, ins+n*K*H*W, K*H*W, outs+n*K*H*W, K*H*W,</li>
          <li>                              H, W, sigmargb, sigmaxy);</li>
      </ol>
    </code>
  </pre>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>i call this function in pytorch after wrapping it in python using swig.<br>
this c++ function creates a parallel region. i assume that each thread will deal with a sample in the minibatch in the lop.</p>
<p>this is actually fast (70ms).<br>
when using ddp+2gpus, the multi-threadins does not seem to work (call 500ms == time when using only 1 thread).</p>
<p>the maximum threads on that machine is 48.<br>
batch size 32.</p>
<p>any idea why?</p>
<p>thank you very much for your help!</p>

username: ptrblck
cooked: <p>I’m not entirely sure, if you are seeing a warning or are setting the OMP threads too late, but <a href="https://github.com/pytorch/pytorch/blob/65e6194aeb3269a182cfe2c05c122159da12770f/torch/distributed/run.py#L596-L608">here</a> <code>omp_num_threads</code> will be set to <code>1</code>.<br>
Could you check, if this could the also the root cause for your observation?</p>

username: sbelharbi
cooked: <p>you are so right!!!<br>
distributed turned off the multithreading and the c++ function was too late when it asks for the max nbr threads which will be 1.</p>
<p>– begin side question<br>
side question: how to ask ddp to log into a specific file?<br>
the dd logging is in terminal which makes catching warnings difficult.<br>
there are a lot of messages but they are lost.<br>
is there a way to ask ddp to write logs in a specific file?<br>
or can we properly manipulate the instance of log here to do that? thanks<br>
the first time i used ddp, it starts throwing logs in terminal which is unpractical for debug.<br>
there should be a way to tell ddp to log into a file.<br>
the <a href="https://pytorch.org/docs/stable/distributed.html" rel="noopener nofollow ugc">doc1</a> and <a href="https://pytorch.org/docs/stable/generated/torch.nn.parallel.DistributedDataParallel.html" rel="noopener nofollow ugc">doc2</a> do not seem to cover this.<br>
i didnt investigate further as other things have more priority.<br>
thanks</p>
<aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/pytorch/blob/65e6194aeb3269a182cfe2c05c122159da12770f/torch/distributed/run.py#L325">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/blob/65e6194aeb3269a182cfe2c05c122159da12770f/torch/distributed/run.py#L325" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/65e6194aeb3269a182cfe2c05c122159da12770f/torch/distributed/run.py#L325" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/65e6194aeb3269a182cfe2c05c122159da12770f/torch/distributed/run.py#L325</a></h4>



  <pre class="onebox">    <code class="lang-py">
      <ol class="start lines" start="315" style="counter-reset: li-counter 314 ;">
          <li>import torch</li>
          <li>from torch.distributed.argparse_util import check_env, env</li>
          <li>from torch.distributed.elastic.multiprocessing import Std</li>
          <li>from torch.distributed.elastic.multiprocessing.errors import record</li>
          <li>from torch.distributed.elastic.rendezvous.utils import _parse_rendezvous_config</li>
          <li>from torch.distributed.elastic.utils import macros</li>
          <li>from torch.distributed.elastic.utils.logging import get_logger</li>
          <li>from torch.distributed.launcher.api import LaunchConfig, elastic_launch</li>
          <li>
          </li>
<li>
          </li>
<li class="selected">log = get_logger()</li>
          <li>
          </li>
<li>
          </li>
<li>def get_args_parser() -&gt; ArgumentParser:</li>
          <li>    """Helper function parsing the command line options."""</li>
          <li>
          </li>
<li>    parser = ArgumentParser(description="Torch Distributed Elastic Training Launcher")</li>
          <li>
          </li>
<li>    #</li>
          <li>    # Worker/node size related arguments.</li>
          <li>    #</li>
      </ol>
    </code>
  </pre>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>–end side question</p>
<p>so, now i configure <code>export OMP_NUM_THREADS=32</code> before running and the runtime is back to 70ms with ddp+2gpus!!! this is cool!<br>
thank you very much! this is a life saver!</p>
<p>also, i was getting this warning which is impossible to read because it is printed on terminal!</p>
<pre><code class="lang-auto">*****************************************
Setting OMP_NUM_THREADS environment variable for each process to 
be 1 in default, to avoid your system being overloaded, please further tune 
the variable for optimal performance in your application as needed.
*****************************************
</code></pre>
<p>which explains everything!</p>
<p>also, i am using <code>distributed.launch</code>  which is explains why i am getting this warning</p>
<pre><code class="lang-auto">The module torch.distributed.launch is deprecated and going to be removed
 in future.Migrate to torch.distributed.run
</code></pre>
<p>in the <a href="https://github.com/pytorch/examples/blob/master/distributed/ddp/README.md" rel="noopener nofollow ugc">examples</a> they provided, they use <code>launch</code>. i should probably switch to <code>run</code>, it seems more up to date.<br>
also, they used launch in <a href="https://pytorch.org/docs/stable/distributed.html#distributed-basics" rel="noopener nofollow ugc">this</a> under <code>Launch utility</code>. probably the doc needs to be updated probably in the next release. i am using pytorch 1.9.0.</p>
<p>again, thank you so much! this was very helpful!</p>

username: ptrblck
cooked: <p>The linked line of code sounds right and you could try to set the <code>--log_dir</code> as described in <a href="https://github.com/pytorch/pytorch/blob/57212054179f1623833d46c852d2ac06b063326c/torch/distributed/run.py#L447-L455">this argument</a>.<br>
Good to hear you’ve solved the threading issue.</p>

username: sbelharbi
cooked: <p>thanks. that seems the way to tell run to log in file. thanks<br>
i think i need to go through run.py. its docstring is well documented.<br>
i expect this to be released in the next version because it does not exit in the current doc.</p>
<p>again, thank you very much for your help!!!</p>

username: sbelharbi
cooked: <p>not urgent, but i tried <a href="https://github.com/pytorch/pytorch/blob/a2934b38f8e9264fe24651ca179c281f25b33eac/torch/distributed/run.py#L447" rel="noopener nofollow ugc">loggin</a> into file with ddp.<br>
it worked partially.<br>
something else is still logging into terminal and not file.<br>
<a href="https://discuss.pytorch.org/t/ddp-logging-into-file/131456">here</a> in a separate thread.<br>
this may be still under dev.</p>
<p>thanks</p>