Multi-GPUs training, inbalance gpu memory, what to do with loss function

username: isalirezag
cooked: <p>I have been using pytorch for a long time, but I still could not find a clear solusion for the problem of multigpu training.<br>
Can someone please help me out.<br>
When we have multiple gpu and large batch size I do the following<br>
<code>net = nn.DataParallel(net)</code><br>
and it simply transfer my model to parallel. Nice!<br>
But what should I do for optimization part? I notice something while using multi gpu that it was not there for when I had just one gpu…</p>
<ol>
<li>the memory for gpus is not balance, usually one of the gpus are having more memory</li>
<li>the performance drops! why? I dont know</li>
</ol>
<p>I think I am still not clear what should I do for <code>criterion</code> and loss function when I am using multiple gpu.<br>
I see pytorch added a few more tutorial there but they are not helping me.<br>
I am an example person, I understand things when I see them in example.<br>
Here, I copy paste the example that is provided by pytorch for training a clasifier.<br>
I modify it to be run on multiple gpus.<br>
But I still have the same issues.<br>
Can someone help me what I am missing and how I should fix it?<br>
Thanks a lot!</p>
<pre><code class="lang-auto">import torch
import torchvision
import torchvision.transforms as transforms

transform = transforms.Compose(
    [transforms.ToTensor(),
     transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])

trainset = torchvision.datasets.CIFAR10(root='./data', train=True,
                                        download=True, transform=transform)
trainloader = torch.utils.data.DataLoader(trainset, batch_size=4,
                                          shuffle=True, num_workers=2)

testset = torchvision.datasets.CIFAR10(root='./data', train=False,
                                       download=True, transform=transform)
testloader = torch.utils.data.DataLoader(testset, batch_size=4,
                                         shuffle=False, num_workers=2)

classes = ('plane', 'car', 'bird', 'cat',
           'deer', 'dog', 'frog', 'horse', 'ship', 'truck')


import torch.nn as nn
import torch.nn.functional as F


class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(3, 6, 5)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.fc1 = nn.Linear(16 * 5 * 5, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)

    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = x.view(-1, 16 * 5 * 5)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x


net = Net().cuda()
net = nn.DataParallel(net)

import torch.optim as optim

criterion = nn.CrossEntropyLoss().cuda()
optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9)


for epoch in range(2):  # loop over the dataset multiple times

    running_loss = 0.0
    for i, data in enumerate(trainloader, 0):
        # get the inputs; data is a list of [inputs, labels]
        inputs, labels = data 
        inputs, labels = inputs.cuda(), labels.cuda()

        # zero the parameter gradients
        optimizer.zero_grad()

        # forward + backward + optimize
        outputs = net(inputs)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()

        # print statistics
        running_loss += loss.item()
        if i % 2000 == 1999:    # print every 2000 mini-batches
            print('[%d, %5d] loss: %.3f' %
                  (epoch + 1, i + 1, running_loss / 2000))
            running_loss = 0.0

print('Finished Training')
</code></pre>

username: ptrblck
cooked: <ol>
<li>
<p>Have a look at <a class="mention" href="/u/thomas_wolf">@Thomas_Wolf</a>’s <a href="https://medium.com/huggingface/training-larger-batches-practical-tips-on-1-gpu-multi-gpu-distributed-setups-ec88c3e51255" rel="nofollow noopener">blog post</a>, where he explains the memory usage of the different devices and how to reduce the imbalance.</p>
</li>
<li>
<p>Do you mean the performance regarding the model accuracy or throughput?</p>
</li>
</ol>

username: isalirezag
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<ol>
<li>I will take a look,</li>
<li>yes I mean in terms of accuracy.</li>
</ol>
<p>In that blog I saw it says if I use dataparallel I should do <code>loss.mean().backward()</code>, is it gonna be wrong if i do <code>loss.backward()</code> instead?</p>
<p>I used the guide in the <a class="mention" href="/u/thomas_wolf">@Thomas_Wolf</a> blog but it gave me error.</p>
<p>Here is my code:</p>
<pre><code class="lang-auto">
import torch
import torchvision
import torchvision.transforms as transforms
from parallel import DataParallelModel, DataParallelCriterion

transform = transforms.Compose(
    [transforms.ToTensor(),
     transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])

trainset = torchvision.datasets.CIFAR10(root='./data', train=True,
                                        download=True, transform=transform)
trainloader = torch.utils.data.DataLoader(trainset, batch_size=4,
                                          shuffle=True, num_workers=2)

testset = torchvision.datasets.CIFAR10(root='./data', train=False,
                                       download=True, transform=transform)
testloader = torch.utils.data.DataLoader(testset, batch_size=4,
                                         shuffle=False, num_workers=2)

classes = ('plane', 'car', 'bird', 'cat',
           'deer', 'dog', 'frog', 'horse', 'ship', 'truck')


import torch.nn as nn
import torch.nn.functional as F


class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(3, 6, 5)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.fc1 = nn.Linear(16 * 5 * 5, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)

    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = x.view(-1, 16 * 5 * 5)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x


net = Net().cuda()
parallel_model = DataParallelModel(net)

import torch.optim as optim

criterion = nn.CrossEntropyLoss().cuda()
optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9)

parallel_loss = DataParallelCriterion(criterion)

for epoch in range(2):  # loop over the dataset multiple times

    running_loss = 0.0
    for i, data in enumerate(trainloader, 0):
        # get the inputs; data is a list of [inputs, labels]
        inputs, labels = data 
        inputs, labels = inputs.cuda(), labels.cuda()

        # zero the parameter gradients
        optimizer.zero_grad()

        # forward + backward + optimize
        outputs = parallel_model(inputs)
        loss = parallel_loss(outputs, labels)
        loss.sum().backward()
        optimizer.step()

        # print statistics
        running_loss += loss.item()
        if i % 2000 == 1999:    # print every 2000 mini-batches
            print('[%d, %5d] loss: %.3f' %
                  (epoch + 1, i + 1, running_loss / 2000))
            running_loss = 0.0

print('Finished Training')
</code></pre>
<p>and I got this error:</p>
<p><code>ValueError: only one element tensors can be converted to Python scalars</code></p>

username: PistonY
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>  Hello,I read the blog and ZhangHang’s code.<br>
Does Pytorch suppory <code>computing the loss in a parallel fashion</code> now?</p>

username: ptrblck
cooked: <p>The error comes most likely from</p>
<pre><code class="lang-python">running_loss += loss.item()
</code></pre>
<p>as <code>loss</code> doesn’t seem to be a scalar.<br>
You could uncomment one of <a href="https://gist.github.com/thomwolf/7e2407fbd5945f07821adae3d9fd1312#file-parallel-py-L160" rel="nofollow noopener">these lines</a> to reduce the loss.</p>
<p><a class="mention" href="/u/pistony">@PistonY</a> <a href="https://gist.github.com/thomwolf/7e2407fbd5945f07821adae3d9fd1312#file-parallel-py" rel="nofollow noopener">These scripts</a> of <a class="mention" href="/u/thomas_wolf">@Thomas_Wolf</a> provide this functionality.</p>

username: isalirezag
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> should i get the same performance for both single gpu and multi-gpu training?</p>
<p>hmmmm not sure what does it mean to reduce the loss… (?)<br>
my loss is like this when I print it [1.255, 54.2] (I have two gpus for this case)<br>
<a class="mention" href="/u/thomas_wolf">@Thomas_Wolf</a> can you help please?</p>

username: ptrblck
cooked: <p>Ideally your performance should scale with the number of GPUs.<br>
What kind of performance do you measure using multiple GPUs vs. single one?</p>
<aside class="quote no-group" data-username="isalirezag" data-post="6" data-topic="52886">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/i/fbc32d/48.png" class="avatar"> isalirezag:</div>
<blockquote>
<p>not sure what does it mean to reduce the loss</p>
</blockquote>
</aside>
<p>You could calculate the mean or sum of the loss and call <code>backward()</code> on the result.</p>

username: isalirezag
cooked: <aside class="quote no-group">
<blockquote>
<p>Ideally your performance should scale with the number of GPUs.</p>
</blockquote>
</aside>
<p>it was a classification task, the performance got worse with increasing gpus which was not making sense to me.</p>
<aside class="quote no-group">
<blockquote>
<p>You could calculate the mean or sum of the loss and call <code>backward()</code> on the result.</p>
</blockquote>
</aside>
<p>I did do that, in the code that i posted I used  <code>        loss.sum().backward()</code>, isnot that what you mean?</p>

username: isalirezag
cooked: <p>i still cannot make it work <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"></p>

username: Alireza_Ahmadi
cooked: <p>I have similar problem which is as follows:</p>
<p>I have a Unet structure which I try to feed back some feature maps, and I am using multi-gpu<br>
when I try to run my code with only one gpu core it throws this error</p>
<p>RuntimeError: Trying to backward through the graph a second time, but the buffers have already been freed. Specify retain_graph=True when calling backward the first time.</p>
<p>but when I run it with two cores it goes fin.<br>
here how I use the backward</p>
<p>curr_loss = self.criterion(smax_outputs, label)</p>
<pre><code>    if self.gpustat['n_gpus'] &gt;= 2:
      Dev_Inc = max(self.cfg['TRAIN_BATCH_SIZE'] // self.gpustat['n_gpus'], 1)
      idx = torch.ones(Dev_Inc).cuda()
      curr_loss.backward(idx)
    else:
      curr_loss.backward()
    
    self.optimizer.step()
    # zero the parameter gradients
    self.optimizer.zero_grad()
</code></pre>
<p>it is not a problem of memory…</p>

username: ptrblck
cooked: <p>The root cause of this error is unfortunately not visible in this code snippet.<br>
Could you post a minimal executable code snippet to reproduce this error with your model and random data, so that we could have a look?</p>