What to send to GPU?

PhysicsIsFun: <p>Hello there,</p>
<p>I am a bit confused about what to send to the GPU.</p>
<p>Take a look at these routines:</p>
<pre><code class="lang-auto">def train(model, train_loader, datasize, lr, weight_decay, alpha, epoch):
    model.train()
    for batch_idx, (data, target) in enumerate(train_loader):
        data, target = data.to(device), target.to(device)
        model.zero_grad()
        output = model(data)
        loss = F.nll_loss(output, target)*datasize
        loss.backward()
        update_params(model, lr, weight_decay, alpha)

def update_params(model, lr, weight_decay, alpha):
    for p in model.parameters():
        if not hasattr(p,'buf'):
            p.buf = torch.randn(p.size()).to(device)*np.sqrt(lr) 
        d_p = p.grad.data
        d_p.add_(weight_decay, p.data)
        eps = torch.randn(p.size()).to(device)
        buf_new = (1-alpha)*p.buf - lr*d_p + (2.0*lr*alpha)**.5*eps
        p.data.add_(buf_new)
        p.buf = buf_new

</code></pre>
<p>In the train-routine, are the “output” and “loss” variables automatically created on the GPU?</p>
<p>In the “update” routine, are the “lr” and “alpha” constants automatically send to the GPU as well?</p>
<p>What if I have a code snipped like this where “self.model” is already on the GPU:</p>
<pre><code class="lang-auto">        loss_train = np.zeros(self.epochs+1)                 # store loss
        accu_train = np.zeros(self.epochs+1)                 # store accuracies
        loss_test = np.zeros(self.epochs+1)
        accu_test = np.zeros(self.epochs+1)
        
        (loss_train[0], accu_train[0]) = self.model.evaluate(self.train_loader)
        (loss_test[0], accu_test[0]) = self.model.evaluate(self.test_loader) 
</code></pre>
<p>Will the loss and accu arrays be on GPU as well? Or will the results of the right-hand side be sent from GPU to CPU first?</p>
<p>Cheers!</p>

ptrblck: <aside class="quote no-group" data-username="PhysicsIsFun" data-post="1" data-topic="163574">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/p/f6c823/48.png" class="avatar"> PhysicsIsFun:</div>
<blockquote>
<p>In the train-routine, are the “output” and “loss” variables automatically created on the GPU?</p>
</blockquote>
</aside>
<p>Yes, if the model parameters and the input used to calculate the <code>output</code> were pushed to the GPU.</p>
<aside class="quote no-group" data-username="PhysicsIsFun" data-post="1" data-topic="163574">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/p/f6c823/48.png" class="avatar"> PhysicsIsFun:</div>
<blockquote>
<p>In the “update” routine, are the “lr” and “alpha” constants automatically send to the GPU as well?</p>
</blockquote>
</aside>
<p>Yes, scalar values are automatically pushed to the GPU if needed.</p>
<aside class="quote no-group" data-username="PhysicsIsFun" data-post="1" data-topic="163574">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/p/f6c823/48.png" class="avatar"> PhysicsIsFun:</div>
<blockquote>
<p>What if I have a code snipped like this where “self.model” is already on the GPU:</p>
</blockquote>
</aside>
<p>It depend on the implementation of the <code>evaluate</code> method which isn’t a built-in PyTorch method of a module.</p>

PhysicsIsFun: <p>Thank you ptrblck!</p>
<p>The model with the <code>evaluate</code> function is given by:</p>
<pre><code class="lang-auto">class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.fc1 = nn.Linear(28*28, 500)
        self.fc2 = nn.Linear(500, 256)
        self.fc3 = nn.Linear(256, 10)

    def forward(self, x):
        x = x.view(-1, 28*28)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        return F.log_softmax(self.fc3(x), dim=1)
    
    def evaluate(self, data_loader):
        self.eval()
        loss = 0
        correct = 0
        criterion = nn.NLLLoss(reduction='sum')
        with torch.no_grad():
            for data, target in data_loader:
                data, target = data.to(device), target.to(device)
                output = self.forward(data)
                loss += criterion(output, target).data.item()             # sum up batch loss
                pred = output.data.max(1, keepdim=True)[1]                # get the index of the max log-probability
                correct += pred.eq(target.data.view_as(pred)).cpu().sum() # does cpu make sense?
    
        loss /= len(data_loader.dataset)
        accu = 100. * correct / len(data_loader.dataset)
        
        return (loss, accu)
</code></pre>
<p>So if I call it via</p>
<pre><code class="lang-auto">(loss_train[0], accu_train[0]) = self.model.evaluate(self.train_loader)
</code></pre>
<p>will the arrays on the left be on GPU? Do questions like these even matter?</p>

ptrblck: <aside class="quote no-group" data-username="PhysicsIsFun" data-post="3" data-topic="163574">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/p/f6c823/48.png" class="avatar"> PhysicsIsFun:</div>
<blockquote>
<p>will the arrays on the left be on GPU?</p>
</blockquote>
</aside>
<p>No, since <code>loss</code> is a Python scalar value created via <code>item()</code> while <code>accu</code> was created from <code>correct</code> which was explicitly pushed to the CPU:</p>
<pre data-code-wrap="python"><code class="lang-python">correct += pred.eq(target.data.view_as(pred)).cpu().sum() # does cpu make sense?
</code></pre>

PhysicsIsFun: <p>Sorry I am still unclear… <code>loss</code> and <code>correct</code> were created via</p>
<pre><code class="lang-auto">loss = 0
correct = 0

</code></pre>
<p>so they were not explicitly pushed to the GPU, but they are created in member function <code>evaluate</code> of an object that was pushed to the GPU. From that reasoning, it would not be clear to me what happens… You seem to say that they are not created on the GPU. OK, then moving on.<br>
In the following snippet, we have</p>
<pre><code class="lang-auto">loss += criterion(output, target).data.item()            
pred = output.data.max(1, keepdim=True)[1]              
correct += pred.eq(target.data.view_as(pred)).cpu().sum()
</code></pre>
<p>The <code>item( )</code> function has no influence on the device I assume. The <code>cpu()</code> function of course does. I assume that without the<code> cpu( )</code>, the expression</p>
<pre><code class="lang-auto">pred.eq(target.data.view_as(pred)).cpu().sum()
</code></pre>
<p>would be created on the GPU as <code>target</code> and <code>pred</code> are on the GPU, no?</p>
<p>Is there a detailed documentation somewhere that explains when stuff is automatically pushed to the GPU? I feel like not understanding that in detail is the cause of many bottlenecks in practice…</p>

ptrblck: <aside class="quote no-group" data-username="PhysicsIsFun" data-post="5" data-topic="163574">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/p/f6c823/48.png" class="avatar"> PhysicsIsFun:</div>
<blockquote>
<p>From that reasoning, it would not be clear to me what happens… You seem to say that they are not created on the GPU. OK, then moving on.</p>
</blockquote>
</aside>
<p>No, that’s not what I explained. The intermediate tensors were created in the GPU (assuming <code>device</code> is pointing to a GPU), but you are explicitly moving the results back to the CPU via the <code>item()</code> and <code>cpu()</code> calls. Even if the tensors were on the GPU previously, you are moving them to the CPU so that <code>loss</code> and <code>accu</code> are both on the CPU.</p>
<aside class="quote no-group" data-username="PhysicsIsFun" data-post="5" data-topic="163574">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/p/f6c823/48.png" class="avatar"> PhysicsIsFun:</div>
<blockquote>
<p>The <code>item( )</code> function has no influence on the device I assume.</p>
</blockquote>
</aside>
<p>That’s wrong, since <code>item()</code> creates a Python scalar which as to be stored on the CPU.</p>
<aside class="quote no-group" data-username="PhysicsIsFun" data-post="5" data-topic="163574">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/p/f6c823/48.png" class="avatar"> PhysicsIsFun:</div>
<blockquote>
<p>would be created on the GPU as <code>target</code> and <code>pred</code> are on the GPU, no?</p>
</blockquote>
</aside>
<p>Yes, that’s correct. The <code>cpu()</code> call moves the result tensor to the CPU.</p>

PhysicsIsFun: <p>And if I only have</p>
<pre><code class="lang-auto">loss = 0
correct = 0
</code></pre>
<p>in the <code>evaluate</code> function,<br>
this would be normal Python scalars on the CPU, even if the object that calls <code>evaluate</code> is on the GPU?</p>

ptrblck: <p>Yes, both variables would be plain Python integers on the CPU, but depending on the type of the object of the later operation you would override these variables and could create tensors as seen here:</p>
<pre><code class="lang-python">loss = 1
loss += torch.ones(1, device='cuda')
print(loss)
# tensor([2.], device='cuda:0')
</code></pre>

PhysicsIsFun: <p>Thank you ptrblck,</p>
<p>I feel more confident now.</p>