Pytorch multi-worker dataloader runs in parallel with training code?

username: rui_zhang_331
cooked: <p>Hi All</p>
<p>we have dataloader and training code works like this way</p>
<pre><code class="lang-auto">for fi, batch in enumerate(my_data_loader):
     train()
</code></pre>
<p>and in our dataloader, we have define some <code>collate_fn</code> to <code>cook_data</code></p>
<pre><code class="lang-auto">DataLoader(my_dataset,
                                        num_workers=config['num_dataloader_worker'],
                                        batch_size=config['dataloader_batch_size'],
                                        timeout=600,
                                        collate_fn=cook_data
                                        )
</code></pre>
<p>My question here is when training is running,  can data loader in parallel run in background to do like <code>cook_data</code>, or each “process” will first load/cook data , then run training, so during training, this particular process is basically blocking waiting there?</p>

username: ptrblck
cooked: <p>The <code>DataLoader</code> will use multiprocessing to create multiple workers, which will load and process each data sample and add the batch to a queue. It should thus not be blocking the training as long as the queue is filled with batches.</p>

username: rui_zhang_331
cooked: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a>, so if the training will be different processes than multiprocessing dataloader, right? from your description, if training is slower than dataloading, then basically we should get continuous training and loading time will he shaded?</p>
<p>Also if I use Data parallel, and based on understanding data parallel is using multi threading, so how this multi threading data parallel will work with multi process data loader? still the same way multi process data loader loads the data into queue, and training process(a different process) spin multi threads according to multi GPU to train ?</p>

username: ptrblck
cooked: <p>Yes, the main process would execute the training loop, while each worker will be spawned in a new process via multiprocessing. <code>nn.DataParallel</code> and the <code>DataLoader</code> do not interfere with each other.<br>
Also yes, if the loading pipeline is faster then the training, the data loading time would be “hidden”.</p>

username: rui_zhang_331
cooked: <p>Thanks, we test our dataloading without training and it is very fast. But we also put some timestamp on our code and found training time is only part of total time, is that because GPU is aync and our time recoding may not be that accurate?<br>
</p><aside class="onebox allowlistedgeneric">
  <header class="source">
      <a href="https://pytorch.org/docs/stable/notes/cuda.html#asynchronous-execution" target="_blank" rel="noopener nofollow ugc">pytorch.org</a>
  </header>
  <article class="onebox-body">
    <img src="" class="thumbnail" width="" height="">

<h3><a href="https://pytorch.org/docs/stable/notes/cuda.html#asynchronous-execution" target="_blank" rel="noopener nofollow ugc">CUDA semantics — PyTorch 1.7.0 documentation</a></h3>



  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<pre><code class="lang-auto">start_time = datetime.now()
for for loo in range(0, config['epochs']):
    for fi, batch in enumerate(my_data_loader):
        train_time = datetime.now()
        train()
        train_endtime = datetime.now()
total_endtime = datetime.now()
</code></pre>

username: ptrblck
cooked: <p>CUDA operations are asynchronous, so you won’t capture their runtime and it will be accumulated in the next blocking operation.<br>
You can profile the complete code e.g. with Nsight Systems and check the timeline to narrow down the bottleneck, if your current profiling with timers isn’t giving enough information (or use the PyTorch profiler and create the timeline output).</p>

username: rui_zhang_331
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>, I did some test for dataloader and pipeline, the test looks like this</p>
<ul>
<li>Experiment 1: With preload data, data pre load into memory</li>
</ul>
<pre><code class="lang-auto">replay_mem = {}
for fi, batch in enumerate(my_data_loader):
      replay_mem[fi] = batch

# Training with all data in memory

for i in range(0,epoch)
    for fi, batch in enumerate(replay_memory.items()):
     Train(batch)
</code></pre>
<ul>
<li>Experiment 2: Without preload data, the data is loaded via dataloader</li>
</ul>
<pre><code class="lang-auto">for i in range(0,epoch)
    for fi, batch in enumerate(ny_data_loader):
     Train(batch)
</code></pre>
<p>The experiment 1 has less time, and time diff compared with experiment 2 is about the data loading time(if I just test the data loading). I imagine if we have pipeline that training just consumes multi worker data loader, if training time is higher than multi worker data loader, so data loader time will be completely shaded? (data loading is always faster and thus training always have data)?</p>

username: ptrblck
cooked: <p>Yes, if the model training takes more time than loading and processing the next batch, the data loading time will be hidden and comes “for free” (The next epoch would create new workers, which would have to start creating new batches in the default setup. You could use <code>persistent_workers=True</code> to avoid this behavior).</p>