Total number of processes and threads created using nn.distributed.parallel

joshua_mora: <p>Under to the context of training using python front end.</p>
<p>Where could I find some information about the total number of processes and threads when using nn.distributed.parallel module ?</p>
<p>If I have a simple neural network (eg. MNIST) and I do distributed data parallelism where I assign 1 process per GPU, and I have both training and eval going on and a dataloader with 1 worker, should I have only 3 processes per GPU: 1 main process (the training one) that spawns eval process and dataloader process (total of 3 processes). Then within the main process: a thread for scheduling work, a thread for forward, a thread for backward, a thread to deal with eval process, a thread to deal with dataloader, a thread for cache manager. That is 6 threads. When profiling I get to see several mode. Is there any document where I can get that info ? Also if BWD is consuming what FWD is producing, is there a way I can “merge” both threads of FWD and BWD in a single thread ? Is there also a way to not dealloc objects from the cache allocator if the number of objects (tensors, model) remains the same from iteration to iteration, so I can avoid the expensive mmap/munmap ?</p>
<p>Thanks in advance.</p>

osalpekar: <p>In terms of the total number of processes, the <code>num_workers</code> argument you pass to the <code>DataLoader</code> class determines the number of subprocess it uses (0 means the dataloader uses the main process). Here is some documentation: <a href="https://pytorch.org/docs/1.1.0/_modules/torch/utils/data/dataloader.html" rel="nofollow noopener">https://pytorch.org/docs/1.1.0/_modules/torch/utils/data/dataloader.html</a>.</p>
<p>For the number of threads, this varies based on the communication backend you use (which is passed to <code>init_process_group</code>). For example the gloo backend uses 2 threads per device: <a href="https://github.com/pytorch/pytorch/blob/master/torch/csrc/distributed/c10d/init.cpp#L565" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/master/torch/csrc/distributed/c10d/init.cpp#L565</a>.</p>

joshua_mora: <p>Thanks for the quick reply.</p>
<p>I am using NCCL backend. I do not see how many threads are being passed to create the group. at <a href="https://github.com/pytorch/pytorch/blob/master/torch/lib/c10d/ProcessGroupNCCL.cpp" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/master/torch/lib/c10d/ProcessGroupNCCL.cpp</a><br>
Is there a knob to control that ?</p>
<p>I am using dataloder with num_workers set to 1 (so main process spawns a separate process ?)<br>
Based on the information on  <a href="https://pytorch.org/docs/1.1.0/_modules/torch/utils/data/dataloader.html" rel="nofollow noopener">https://pytorch.org/docs/1.1.0/_modules/torch/utils/data/dataloader.html</a>.<br>
At each iteration the dataloader process is created and destroyed (if num_workers!=0) which has some overhead ?<br>
Can we keep the processes (depending on how many samples within the batch you want to work concurrently) across iterations so we do not incur into that overhead ?</p>
<p>I am basically trying to prune the number of processes and threads, while I understand I may restrict generality but I am trying to speed up the execution when I am CPU bound.</p>
<p>Thanks in advance.</p>

osalpekar: <aside class="quote no-group" data-username="joshua_mora" data-post="3" data-topic="71043" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/j/e19b73/48.png" class="avatar"> joshua_mora:</div>
<blockquote>
<p>I am using NCCL backend. I do not see how many threads are being passed to create the group. at <a href="https://github.com/pytorch/pytorch/blob/master/torch/lib/c10d/ProcessGroupNCCL.cpp" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/blob/master/torch/lib/c10d/ProcessGroupNCCL.cpp</a><br>
Is there a knob to control that ?</p>
</blockquote>
</aside>
<p>The number of threads is currently not tunable by the user, but we’re considering making this possible in a future release.</p>
<aside class="quote no-group quote-modified" data-username="joshua_mora" data-post="3" data-topic="71043" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/j/e19b73/48.png" class="avatar"> joshua_mora:</div>
<blockquote>
<p>I am using dataloder with num_workers set to 1 (so main process spawns a separate process ?)<br>
Based on the information on  <a href="https://pytorch.org/docs/1.1.0/_modules/torch/utils/data/dataloader.html" class="inline-onebox" rel="noopener nofollow ugc">torch.utils.data.dataloader — PyTorch master documentation</a>.<br>
At each iteration the dataloader process is created and destroyed (if num_workers!=0) which has some overhead ?<br>
Can we keep the processes (depending on how many samples within the batch you want to work concurrently) across iterations so we do not incur into that overhead ?</p>
</blockquote>
</aside>
<p>Right, <code>num_workers=1</code> would spawn a separate process. Here’s an issue tracking the discussion around keeping subprocesses alive across iterations (with a patch that should make this possible):</p><aside class="onebox githubissue" data-onebox-src="https://github.com/pytorch/pytorch/issues/15849">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/issues/15849" target="_blank" rel="noopener nofollow ugc">github.com/pytorch/pytorch</a>
  </header>

  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewBox="0 0 14 16" aria-hidden="true"><path fill-rule="evenodd" d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/issues/15849" target="_blank" rel="noopener nofollow ugc">DataLoader with option to re-use worker processes</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2019-01-09" data-time="00:21:24" data-timezone="UTC">12:21AM - 09 Jan 19 UTC</span>
      </div>


      <div class="user">
        <a href="https://github.com/dashesy" target="_blank" rel="noopener nofollow ugc">
          <img alt="dashesy" src="https://avatars.githubusercontent.com/u/873905?v=4" class="onebox-avatar-inline" width="20" height="20">
          dashesy
        </a>
      </div>
    </div>

    <div class="labels">
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          high priority
        </span>
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          feature
        </span>
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          module: dataloader
        </span>
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          triaged
        </span>
    </div>
  </div>
</div>

  <div class="github-row">
    <p class="github-body-container">## 🚀 Feature
Currently after an epoch is ended dataloader spawns a new process <span class="show-more-container"><a href="" rel="noopener" class="show-more">…</a></span><span class="excerpt hidden">to read data. This means if processes have cached some internal state (db connection, indexing, ...) they will be lost and the new process will have the overhead of creating the connection or indexing.

## Motivation

I have a custom dataset (that indexes data for fast retrieval) and only part of the dataset is used in training. If I have a training with longer epochs, training will be faster than shorter epochs because when an epoch ends a new process will have to re-read and index the custom dataset.

## Pitch

A flag to tell dataloader to re-use the worker processes. If the process pool has the re-use option ([Loky from joblib](https://github.com/tomMoral/loky) does) that would be enough.

## Alternatives

Documentation should specify this issue. Option to use [Loky](https://github.com/tomMoral/loky) for all multi-processing backend will be also great.

## Additional context

Started a discussion [here](https://discuss.pytorch.org/t/dataloader-re-use-worker-processes/34071)


cc @ezyang @gchanan @SsnL</span></p>
  </div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


joshua_mora: <p>Thanks for the pointer to this discussion on the dataloader.<br>
With respect to the number of threads/processes, I still miss to understand all the other threads being generated.<br>
Is it possible for example to use same thread for backward and forward if they deal with same model and batch, instead of having 2 threads that could assume using different models and samples ?<br>
NCCL process group also accepts the option of size which I am not sure it refers also to the number of threads.<br>
Are there any other hardcoded number of threads that I could reduce ? ( set_num_threads(1)/set_num_interop_threads(1) will not prevent from creating a bunch of threads, larger than 6 per process that deals with each GPU). I have very few cores available per GPU (~4) so I need to restrict the number of threads to what is necessary.</p>
<p>Thanks, again.</p>

osalpekar: <aside class="quote no-group" data-username="joshua_mora" data-post="5" data-topic="71043" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/j/e19b73/48.png" class="avatar"> joshua_mora:</div>
<blockquote>
<p>Is it possible for example to use same thread for backward and forward if they deal with same model and batch, instead of having 2 threads that could assume using different models and samples ?</p>
</blockquote>
</aside>
<p>I’m not sure of any way to coerce forward and backward into using the same thread.</p>
<aside class="quote no-group" data-username="joshua_mora" data-post="5" data-topic="71043" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/j/e19b73/48.png" class="avatar"> joshua_mora:</div>
<blockquote>
<p>NCCL process group also accepts the option of size which I am not sure it refers also to the number of threads.</p>
</blockquote>
</aside>
<p>That size actually refers to world_size, which is the total number of ranks in your job</p>
<aside class="quote no-group" data-username="joshua_mora" data-post="5" data-topic="71043" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/j/e19b73/48.png" class="avatar"> joshua_mora:</div>
<blockquote>
<p>Are there any other hardcoded number of threads that I could reduce ? ( set_num_threads(1)/set_num_interop_threads(1) will not prevent from creating a bunch of threads, larger than 6 per process that deals with each GPU). I have very few cores available per GPU (~4) so I need to restrict the number of threads to what is necessary.</p>
</blockquote>
</aside>
<p>This might provide some more insight into tuning the number of threads: <a href="https://github.com/pytorch/pytorch/issues/16894" class="inline-onebox" rel="noopener nofollow ugc">Number of CPU threads for the python process · Issue #16894 · pytorch/pytorch · GitHub</a>. For example, the OMP_NUM_THREADS env var is used for controlling the number of OpenMP threads for CPU operations and MKL_NUM_THREADS for mkl.</p>

joshua_mora: <p>Thanks <a class="mention" href="/u/osalpekar">@osalpekar</a>.<br>
OpenMP environment variables dont make a difference if torch.set_num_threads is already set.<br>
In fact I use GOMP_CPU_AFFINITY to enforce a particular set of openMP threads to run on specific cores. I did play also with OMP/MKL_DYNAMIC set to false.<br>
I still do not understand though what torch.set_num_threads controls if I end up having 1 thread for FWD and 1 different thread for BWD. And several other threads.<br>
I may think that there is a total amount of work and some “empirical” definition of how many threads to have for certain amount of work. the env var will overwrite that heuristic.<br>
Just trying to find the code where that is precribed/defined.<br>
Are you aware of a document where it is described the architecture in terms of thread/functionality ?<br>
Why would I get &gt; 6 threads if I have torch.set_num_threads set to 1.</p>
<p>Regards.</p>