DataParallel not effective

Andy-jqa: <p>Hi all,<br>
I have a quite large model and need to do data parallel among multiple GPUs.<br>
I used:</p>
<pre><code class="lang-python">model = nn.DataParallel(model)
</code></pre>
<p>And there are three visible GPUs. The GPU usage is:</p>
<pre><code class="lang-bash">+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0     12444      C   python                                     10741MiB |
|    1     12444      C   python                                      4663MiB |
|    2     12444      C   python                                      4633MiB |
+-----------------------------------------------------------------------------+
</code></pre>
<p>GPU1 and GPU2 are not fully utilized, but I cannot increase the batch size because then there will be memory error on GPU0.<br>
Does anyone know how to solve this problem?</p>

soulless: <p>Hello, I am having the same issue. Have you solved it? Thanks</p>

liujie_zhang: <aside class="quote no-group" data-username="Andy-jqa" data-post="1" data-topic="18221">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/e274bd/48.png" class="avatar"> Andy-jqa:</div>
<blockquote>
<p>nn.DataParallel</p>
</blockquote>
</aside>
<p>Hi, Andy-jpa<br>
If the the first gpu (id:0) is occupied  all the time and no more space to use, you could consider to use only 1 and 2 gpu explictly assigned in you code by device_ids.</p>
<p>refer : <a href="https://pytorch.org/tutorials/beginner/former_torchies/parallelism_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Multi-GPU Examples — PyTorch Tutorials 2.1.1+cu121 documentation</a></p>
<p>code like this:</p>
<pre><code class="lang-auto">def data_parallel(module, input, device_ids, output_device=None):
    if not device_ids:
        return module(input)

    if output_device is None:
        output_device = device_ids[0]

    replicas = nn.parallel.replicate(module, device_ids)
    inputs = nn.parallel.scatter(input, device_ids)
    replicas = replicas[:len(inputs)]
    outputs = nn.parallel.parallel_apply(replicas, inputs)
    return nn.parallel.gather(outputs, output_device)
</code></pre>

hashio: <p>Hi I have same issue.<br>
I tried 16GPU.</p>
<p>I think this problem is batch_size need increase to use efficiently multiple GPUs but outputs increase either.<br>
replicas … batch_size / GPUs * input_size<br>
output_device … batch_size * GPUs * output_size<br>
but output_device must have replica in the current DataParallel imprementation.<br>
Therefore I try rewrite DataParallel to output_device avoid from replicas.<br>
In my case, I could increase batch size to output_device GPU use 30GB and each replica GPUs uses 20GB.</p>
<p>But, I’m not sure it’s correct.<br>
I know, it need resolve hide latency to get more efficiency process.<br>
best solution is change to concurrent processing.</p>
<p>I have used PyTorch since last week replace from TensorFlow2.<br>
I want to know best practice of PyTorch.</p>

ptrblck: <p>The fastest and recommended approach is <code>DistributedDataparallel</code> using a single process per GPU as described in <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.parallel.DistributedDataParallel">these docs</a>.</p>

hashio: <p>thanks quick reply<br>
i’ll try it</p>

hashio: <p>DDP is a very good performance and nothing worried for memory management.<br>
thanks for ptrblck and pytorch team</p>