Cpu faster than gpu?

mattinjersey: <p>I am running PyTorch on GPU computer.<br>
Actually I am observing that it runs slightly faster with CPU than with GPU.<br>
About 30 seconds with CPU and 54 seconds with GPU.<br>
Is it possible?<br>
There are some steps where I convert to cuda(), could that slow it down?</p>
<p>Could it be a problem with the computer- it is cloud computer service.</p>
<p>Hard to share my code as it is kind of long and somewhat proprietary.</p>

ptrblck: <p>Could you explain your use case a bit?<br>
If your workload isn’t that big (e.g. small model) this might be the case.<br>
Are you using small batches with very little calculation?</p>

samarth-robo: <p>This can happen when the cost of transferring data between RAM and GPU memory is more than the speedup of parallel computation on the GPU.</p>
<p>One case, as mentioned in <a class="mention" href="/u/ptrblck">@ptrblck</a>’s answer is when your model is quite small. Another case is when you have too many back-and-forth transfers of data in your <code>forward()</code> function. See <a href="https://pytorch.org/docs/stable/notes/cuda.html" rel="nofollow noopener">https://pytorch.org/docs/stable/notes/cuda.html</a> for an example of when data is transferred between RAM and GPU memory.</p>

mattinjersey: <p>Its like a five layer convolutional network with 64 elements per layer, and then I use minibatch size of 1000. Input vector size of 64.<br>
That might qualify as small model.</p>

ptrblck: <p>By 64 elements you mean 64 <code>out_channels/kernels</code>?<br>
Are you using a <code>DataLoader</code>? Do you have any additional transfers in your model as <a class="mention" href="/u/samarth-robo">@samarth-robo</a> asked?</p>

mattinjersey: <p>Not using a dataloader.</p>
<pre><code class="lang-python">class CNN(nn.Module):
    def __init__(self ,NumBins=32):
        self.InNodes=int(NumBins)*2
        self.MediumNode=self.InNodes*2
        super(CNN, self).__init__()
        self.Lin1 = nn.Linear(self.InNodes , self.MediumNode)
        self.Lin2 = nn.Linear(self.MediumNode,  self.MediumNode)
        self.Lin5 = nn.Linear(self.MediumNode, 2)
    def forward(self, input):
        Zoutput = self.Lin1(input)
        Zoutput=F.relu(Zoutput)
        Zoutput = self.Lin2(Zoutput)
        Zoutput=F.relu(Zoutput)
        Zoutput = self.Lin5(Zoutput)
        return Zoutput

</code></pre>

ptrblck: <p>The model itself is faster on my machine using the GPU:</p>
<pre><code class="lang-python">x = torch.randn(1000, 64)
model = CNN()

cpu_times = []

for epoch in range(100):
    t0 = time.perf_counter()
    output = model(x)
    t1 = time.perf_counter()
    cpu_times.append(t1 - t0)

device = 'cuda'
model = model.to(device)
x = x.to(device)
torch.cuda.synchronize()

gpu_times = []
for epoch in range(100):
    torch.cuda.synchronize()
    t0 = time.perf_counter()
    output = model(x)
    torch.cuda.synchronize()
    t1 = time.perf_counter()
    gpu_times.append(t1 - t0)

print('CPU {}, GPU {}'.format(
    torch.tensor(cpu_times).mean(),
    torch.tensor(gpu_times).mean()))
&gt; CPU 0.0018446099711582065, GPU 0.0003588759864214808
</code></pre>
<p>Try to use a <code>DataLoader</code> with <code>pin_memory=True</code> if you are using the GPU, as this will pre-load the next batches to the page-locked memory while the GPU is busy.</p>
<p>What numbers do you get running my script?</p>
<p>PS: I’ve formatted your code. You can add code blocks using three backticks `. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=6" title=":wink:" class="emoji" alt=":wink:"></p>

mattinjersey: <p>thank you for answer.<br>
There is another difference.<br>
I have a generating function<br>
‘’’<br>
aInput = f()<br>
‘’’<br>
that generates data, and then I have a conversion step,<br>
‘’’<br>
aInput=aInput.cuda().<br>
‘’’<br>
Could that slow it down?</p>

ptrblck: <p>Where are you calling this code?<br>
Are you pushing all your data onto the GPU or is it just a batch?<br>
Usually you push the data in the training loop onto the GPU.</p>

mattinjersey: <pre><code class="lang-auto">BigBatchSize=100000
miniBatchSize=1000
BigInput,BigTruth=ComputeResults(BigBatchSize)
for s in range(100):
      xPermute=torch.randperm(BigBatchSize)
      indices=xPermute[0:miniBatchSize]
      sInput=BigInput[indices,:]
      sTruth=BigTruth[indices,:]
      if GPU:
        sInput=sInput.cuda()
        sTruth=sTruth.cuda()
      outGuess=aModel(sInput)</code></pre>

ptrblck: <p>The <code>randperm</code> call is quite expensive. Especially since you only need 1000 indices.<br>
You could use <code>torch.randnint</code>, if duplicates are OK. Otherwise maybe you could permute once outside the loop and then use a “sliding” window getting your mini batch size.<br>
This would affect both the CPU and GPU code.<br>
Were you able to run my small timing script?</p>

samarth-robo: <p><a class="mention" href="/u/mattinjersey">@mattinjersey</a>, it seems to me that the difference between your code and <a class="mention" href="/u/ptrblck">@ptrblck</a>’s code is that the latter only measures the time for computation on the GPU, it does not account for the data transfer time (data transfer happens only once in <a class="mention" href="/u/ptrblck">@ptrblck</a>’s code).</p>
<p>On the other hand, you transfer data to the GPU at every iteration, and hence you are observing the additional time required for that.</p>
<p>Is it possible, as <a class="mention" href="/u/ptrblck">@ptrblck</a> suggested, to write your <code>ComputeResults()</code> function as a <a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.Dataset" rel="nofollow noopener">dataset</a>, so that a <a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.DataLoader" rel="nofollow noopener">dataloader</a> can generate data batches in pin-locked memory (by passing <code>pin_memory=True</code>)? Once the data batch is in pin-locked memory, you can also pass <code>non_blocking=True</code> to the <code>cuda()</code> calls so that data transfer happens asynchronously with computations.</p>
<p>These 2 steps should help amortize the data transfer costs that are slowing you down.</p>

mattinjersey: <p>Yes I confirm the timing results of <a class="mention" href="/u/ptrblck">@ptrblck</a> .<br>
I will try this dataset approach that you have described…<br>
Thx! Matt</p>

LinjX: <p>So, are there any profile tool that detects GPU performance in python?</p>

ptrblck: <p>Maybe <a href="https://pytorch.org/docs/stable/bottleneck.html">torch.utils.bottleneck</a> might be suitable for your profiling.</p>

mattinjersey: <p>also…<br>
GPU-z can check GPU speed for Windows. On Linux,  <code>nvidia-smi -l</code>  will do the trick.</p>

atokta: <p>Hi, I have a similar issue. I am trying to train a simple CNN network using my gpu (gtx 1070). Here is the code:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.nn.functional as F
import numpy as np
import pandas as pd 
from torchvision import datasets, transforms
from torchvision.utils import make_grid
from torch.utils.data import DataLoader
from sklearn.metrics import confusion_matrix
import matplotlib.pyplot as plt
import time
use_cuda = torch.cuda.is_available()
device = torch.device("cuda:0" if use_cuda else "cpu")
## Create transform method 
transform = transforms.ToTensor()

#Download test and train data 
train_data = datasets.MNIST(root='C:\\Users\\aybars\\Google Drive\\pytorch',train=True,download=True,transform=transform)
test_data =datasets.MNIST(root='C:\\Users\\aybars\\Google Drive\\pytorch',train=False,download=True,transform=transform)
 
#create loader object for batch loading

train_loader = DataLoader(train_data,batch_size=10,shuffle=True,pin_memory=True)
test_loader =DataLoader(test_data,batch_size=10,shuffle=True,pin_memory=True)

#create conv layers 
conv1 = nn.Conv2d(1,6,3,1)
conv2 = nn.Conv2d(6,16,3,1)

class ConvolutionalNeuralNetwork(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(1,6,3,1)
        self.conv2 = nn.Conv2d(6,16,3,1)
        self.fc1 = nn.Linear(5*5*16,120)
        self.fc2 = nn.Linear(120,84)
        self.fc3 = nn.Linear(84,10)
        
    def forward(self,X):
        X = F.relu(self.conv1(X))
        X = F.max_pool2d(X,2,2)
        X = F.relu(self.conv2(X))
        X = F.max_pool2d(X,2,2)
        X = X.view(-1,5*5*16)
        X = F.relu(self.fc1(X))
        X = F.relu(self.fc2(X))
        X = F.log_softmax(self.fc3(X),dim=1)
        return X

torch.manual_seed(42)
model = ConvolutionalNeuralNetwork()
model = model.to(device)
print (model)


criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(),lr=0.001)


epoch = 2
train_losses = []
test_losses= []
train_correct=[]
test_correct =[]
t = time.time()
for i in range(epoch):
    trn_correct = 0
    tst_correct = 0
    for b,(X_train,y_train) in enumerate(train_loader):
        X_train=X_train.to(device)
        y_train=y_train.to(device)
        y_pred = model.forward(X_train)
        loss = criterion(y_pred,y_train)
        trn_correct+=(torch.max(y_pred,1)[1]==y_train).sum()
        
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
        
        if b%600==0:
            print(f"Epoch : {i} BATCH: {b} LOSS: {loss.item()}")

    train_losses.append(loss)
    train_correct.append(trn_correct)
    print(f'train_accuracy : {trn_correct.item()/len(train_data)}')
elapsed_time = time.time()-t
print(f'Duration : {elapsed_time}')
</code></pre>
<p>it takes roughly 150 seconds. However when I change</p>
<pre><code class="lang-auto">device = torch.device("cuda:0" if use_cuda else "cpu") 
</code></pre>
<p>to</p>
<pre><code class="lang-auto">device = torch.device("cpu" if use_cuda else "cpu") 
</code></pre>
<p>and setting pin_memory parameters to “False”<br>
without changing anything else it takes 86 seconds.</p>

ptrblck: <p>If you increase the workload (e.g. more filters in the conv layers or more conv layers in general), you should see the benefits of using the GPU. As explained above, tiny workloads might suffer from the overheads of pushing the data to the device as well as the kernel launches.</p>

Duncan_Wood: <p>I used this code exactly and yet my gpu (GTX 2080 with 50th percentile performance) was slower than my cpu (i7-8700) with the results being: CPU 0.0009041459998115897, GPU 0.0022378258872777224. Any reason why this might be? If i increase the data size a lot the GPU begins to clearly outperform but I thought my GPU was pretty good so why wouldn’t I see a performance difference at the same data quantities that your machine does?</p>

ptrblck: <p>I don’t know which code you are using or referring to. In case you are using conv layers, use <code>torch.backends.cudnn.benchmark = True</code>, add some warmup iterations, and profile the code again.<br>
You could also profile the code using the PyTorch profiler or e.g. Nsight Systems to check for potential bottlenecks.</p>