Data Parallelism on single GPU

Ashima_Garg: <p>According to the PyTorch tutorials at <a href="https://pytorch.org/tutorials/beginner/blitz/data_parallel_tutorial.html" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/blitz/data_parallel_tutorial.html</a><br>
which says data parallelism can be done if there are multiple-GPUs, by creating replicas of the model on different GPUs.</p>
<p>Is it possible to use data parallelism on a single GPU device by using more memory on the same device to create replicas of the model and parallelizing the training of different batches on these replicas of the model?</p>
<p>My model is three convolutional layers deep and but I have larger batch size. So, it can parallelize by processing smaller batches in different replicas of the model.</p>
<p>Thanks in advance.</p>

ptrblck: <p>You could try to execute the same script multiple times and check, if the performance really increases.<br>
Besides the GPU memory, the actual computation will use resources on the device and can saturate it completely, so that multiple scripts will enqueue their kernels on the device.</p>
<p>The GPU util output in <code>nvidia-smi</code> gives the percentage of the usage of compute in the last time frame, which might be used as an indicator.</p>

Ashima_Garg: <p>Thanks for your response.<br>
In general, PyTorch does not automatically parallelize the forward pass in a model?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/c/4cb652ac2688a33e462f8a70b16dfbf4785b371d.png" data-download-href="https://discuss.pytorch.org/uploads/default/4cb652ac2688a33e462f8a70b16dfbf4785b371d" title="Computation time"><img src="https://discuss.pytorch.org/uploads/default/original/3X/4/c/4cb652ac2688a33e462f8a70b16dfbf4785b371d.png" alt="Computation time" data-base62-sha1="aWCUAZJ68Wbk4otLkJ4laME3guN" width="690" height="67" data-dominant-color="353630"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Computation time</span><span class="informations">1477×145 8.09 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Can you please check the following results using the autograd profiler.<br>
The computation time of forward pass of through a network of different batch sizes. If it would have been parallelized then for larger batch sizes it could reduce the computation time.</p>
<p>Also, I don’t understand why the CPU time avg and CUDA time avg are almost identical?</p>
<p>Thanks.</p>

ptrblck: <p>PyTorch tries to max out the GPU utilization for the executed operations, such that multiple script executions might not be able to run in parallel.<br>
The CPU time might accumulate the GPU execution time, but I’m not completely sure, as I’m usually either profile operations in isolation (using <code>torch.cuda.synchronize()</code> and a timer) or with <code>nvprof</code>/<code>nsys</code>.</p>

Ashima_Garg: <p>I haven’t used both <code> nvprof/nsys</code> and <code>torch.cuda.synchronize()</code>.</p>
<p>I have just used PyTorch’s autograd profiler with <code>use_cuda = True</code> and <code>torch.autograd.profiler.record_function()</code> over these functions.</p>
<p>Is that not the right way to profile in PyTorch? Also, I didn’t understand the use of <code>nvprof</code> for CUDA events. Can you explain about it?</p>

ptrblck: <p>It might be the right way to profile the code and as I said, I’m not familiar enough with the output and think that the CPU time might be accumulated during the GPU runtime.</p>
<p>I usually use markers and <code>nprof/nsys</code>, as it gives me finer-grained control over the methods I would like to profile.</p>
<p>You could use <a href="https://pytorch.org/docs/stable/cuda.html#nvidia-tools-extension-nvtx">nvtx markers</a> and have a look at <a href="https://gist.github.com/mcarilli/213a4e698e4a0ae2234ddee56f4f3f95">this code snippet</a> for an example use case.<br>
You can find more information about nsight systems <a href="https://docs.nvidia.com/nsight-systems/profiling/index.html#cli-profiling">here</a>.</p>

jon-chuang: <blockquote>
<p>Is it possible to use data parallelism on a single GPU device by using more memory on the same device to create replicas of the model and parallelizing the training of different batches on these replicas of the model?</p>
</blockquote>
<p>Why not just use a larger batch size? DDP is essentially just a larger batch.</p>