DataParallel dosen't work

username: 887574002
cooked: <p>Hi, I am trying to use <code>DataParallel</code> to get ride of out-of-memory that I ran into when I train my code. I read the docs of PyTorch and I found it quite easy. But I do not know for what reason it doesn’t work in my code.<br>
Here is my code. The model is a resnet in 3D.</p>
<pre><code class="lang-auto">    model = generate_resnet3D(conf.model_depth,conf.in_c)
   model = nn.DataParallel(model, device_ids = [0,4,5])
   model.to(device)
 
def train(epoch):
    
    model.train()
    
    # to track the training loss as the model trains
    train_losses = 0
    
    for num,(vol1,vol2) in  enumerate(train_loader):
        
        
        # first we change volums to torch tensors
        vol1 = vol1.to(device)
        vol2 = vol2.to(device)

        
        # next we apply the model to get representations
        
        x1,h1 = model(vol1)
        x2,h2 = model(vol2)
        
        
        print("Outside: input size", vol1.size(),
          "output_size", x1.size())```

I use a batch of size 3. But when I check the output I found that the batch is not split.

```Outside: input size torch.Size([3, 1, 128, 128, 128]) output_size torch.Size([3, 2048, 10, 10, 10])
Outside: input size torch.Size([3, 1, 128, 128, 128]) output_size torch.Size([3, 2048, 10, 10, 10])
Outside: input size torch.Size([3, 1, 128, 128, 128]) output_size torch.Size([3, 2048, 10, 10, 10])
Outside: input size torch.Size([3, 1, 128, 128, 128]) output_size torch.Size([3, 2048, 10, 10, 10])
Outside: input size torch.Size([3, 1, 128, 128, 128]) output_size torch.Size([3, 2048, 10, 10, 10])
Outside: input size torch.Size([3, 1, 128, 128, 128]) output_size torch.Size([3, 2048, 10, 10, 10])```


Here is another example that I checked DataParallel:
```gpu_usage()

model = nn.DataParallel(model,device_ids=[0,1])

device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')

model.to(device)

input = torch.randn(2,1,128,128,128)

input.to(device)

output1, output2 = model(input)

print("Outside: input size", input.size(),
          "output_size", output2.size())

torch.cuda.empty_cache()

sys.stdout.flush()```

The output again wasn't split in two parts!

```| ID | GPU | MEM |
------------------
|  0 |  0% |  7% |
|  1 |  0% |  0% |
|  2 | 84% | 64% |
|  3 | 56% | 50% |
|  4 |  0% |  7% |
|  5 |  0% |  7% |
|  6 | 94% | 99% |
|  7 | 92% | 99% |
Outside: input size torch.Size([2, 1, 128, 128, 128]) output_size torch.Size([2, 2048])```

The first line shows the memory usage. I will be thankful if you help me to fix this problem.</code></pre>

username: ptrblck
cooked: <p><code>nn.DataParallel</code> will split the inputs before passing them to the <code>forward</code> method and will concatenate each output to the expected complete batch output after the <code>forward</code> pass was done.<br>
If you print the shapes outside of the <code>forward</code> method, you won’t see any difference compared to a single GPU run.<br>
To see the split size, you could add the print statements to the <code>forward</code> method directly.</p>

username: 887574002
cooked: <p>But in the second example, as you see I am printing the input before sending it to the forward method, don’t I? It’s the same as example in the PyTorch docs.</p>
<p>And would you please tell me how can I add print statement to the forward method, directly?</p>

username: ptrblck
cooked: <p>This should work and would print the shape of each chunk of the input batch:</p>
<pre><code class="lang-python">def forward(self, x):
    print(x.shape, x.device)
    ...
</code></pre>
<p>If you are printing the tensor outside of the <code>forward</code>, you won’t see any difference to a single GPU run.</p>

username: 887574002
cooked: <p>Ok, thank you. I got it. So, when I use <code>DataPrarallel</code> I shouldn’t encounter the problem of out of memory, right?</p>

username: ptrblck
cooked: <p><code>nn.DataParallel</code> could create a memory imbalance as described in <a href="https://medium.com/huggingface/training-larger-batches-practical-tips-on-1-gpu-multi-gpu-distributed-setups-ec88c3e51255">this blog post</a> with some workarounds.<br>
E.g. if your single GPU run worked with a batch size of 16, <code>nn.DataParallel</code> could yield an OOM for a batch size of 32.</p>
<p>That being said, we generally recommend to use <code>DistributedDataParallel</code> with a single process per device, as this would avoid the memory imbalance and is the fastest approach.</p>

username: JayanthRR
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>, I have a question along similar lines.</p>
<p>I don’t think <code>nn.DistrubtedDataParallel</code>  behaves differently (printing shape before and after forward pass gives the same value i.e. per gpu batch size). What should I do, if I want to concatenate the outputs of the forward passes across all gpus while using <code>nn.DistributedDataParallel</code> ?</p>
<p>Note: I want this as I want to use the model as a feature extractor, and I want to cluster the embeddings (so I need all the data concatenated).</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="JayanthRR" data-post="7" data-topic="88004">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jayanthrr/48/4771_2.png" class="avatar"> JayanthRR:</div>
<blockquote>
<p>I don’t think <code>nn.DistrubtedDataParallel</code> behaves differently</p>
</blockquote>
</aside>
<p>It depends what you are comparing it against.<br>
DDP with multiple devides per process performs similar to <code>nn.DataParallel</code>, DDP with a single device per process does not and is the recommended way. <a href="https://pytorch.org/docs/stable/notes/cuda.html#use-nn-parallel-distributeddataparallel-instead-of-multiprocessing-or-nn-dataparallel">Here</a> is also a short explanation. Also <a href="https://pytorch.org/tutorials/beginner/dist_overview.html">here</a> is a comparison of the distributed backends.</p>
<aside class="quote no-group" data-username="JayanthRR" data-post="7" data-topic="88004">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jayanthrr/48/4771_2.png" class="avatar"> JayanthRR:</div>
<blockquote>
<p>What should I do, if I want to concatenate the outputs of the forward passes across all gpus while using <code>nn.DistributedDataParallel</code> ?</p>
</blockquote>
</aside>
<p>You don’t have to do it manually and the output of the DDP model should already contain all samples. Have a look at <a href="https://pytorch.org/tutorials/intermediate/ddp_tutorial.html">this tutorial</a> for an example.</p>

username: JayanthRR
cooked: <p>Thanks for the reply.  Yes, I am considering the case of single device per process.</p>