Gpu is almost not being used while training but data and model are on device

Jaredeco: <p>My GPU utilization is about 1% while training when I work with an image dataset passed to DataLoader, increasing batch size and num_workers does not help, however when I work with csv data and I do not preprocess it with DataLoader(I pass the whole dataset through model not using batches) it uses GPU and everything works fine but it works only if I make Variable from tensor when I try to put tensor on device(tensor.to(device)) nothing happens and it runs on Cpu. Thanks for help, I have been trying to fix this for a long time.</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.nn.functional as F
from torchvision import transforms
from torchvision import datasets
train_path = r'D:\Datasets\fruits\5857_1166105_bundle_archive\fruits-360\Training'
transform =  transforms.Compose([
    transforms.Resize((32,32)),
    transforms.ToTensor(),
])
train_data = datasets.ImageFolder(train_path, transform=transform)
train_loader = torch.utils.data.DataLoader(
    train_data,
    batch_size=200,
    num_workers=4,
    shuffle=True,
)
class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        self.conv1 = nn.Conv2d(3, 6, 3)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(6, 16, 3)
        self.fc1 = nn.Linear(16*6*6, 64)
        self.fc2 = nn.Linear(64, 131)
    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = x.view(-1, 16*6*6)
        x = F.relu(self.fc1(x))
        x = self.fc2(x)
        return x

device = torch.device("cuda")
net = Model().cuda()
import torch.optim as optim
loss_fn = nn.CrossEntropyLoss()
opt = optim.Adam(net.parameters())
from torch.autograd import Variable
for epoch in range(10):
    for i, data in enumerate(train_loader):
        inputs, labels = data[0], data[1]
        inputs = Variable(inputs).cuda()
        labels = Variable(labels).cuda()
        torch.no_grad()
        out = net(inputs)
        loss = loss_fn(out, labels)
        loss.backward()
        opt.step()
    print(loss.item())
</code></pre>

TheDoctor: <p>What is the resolution of your images? Is it in jpg format? I noticed with my CPU R5 2600X that the cpu was already at its limit decoding the image so i lowered the resolution of the image to 1000x1000 with a external Image Compressor and used the compressed images for the dataset. I aussme your CPU runs jpg decoding for most of the time.</p>

Jaredeco: <p>They are resized to 32*32 pixels, that did not help but i have problem with putting it on gpu.</p>

Nikronic: <p>Hi,</p>
<p>Try to send <code>opt</code> to cuda using <code>.cuda()</code></p>
<aside class="quote no-group" data-username="Jaredeco" data-post="1" data-topic="86686">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jaredeco/48/25963_2.png" class="avatar"> Jaredeco:</div>
<blockquote>
<p><code>opt = optim.Adam(net.parameters())</code></p>
</blockquote>
</aside>
<p>Also, <code>Variable</code> has been removed from PyTorch for few last versions. You can directly send tensors to GPU using <code>inputs = inputs.cuda()</code>.</p>
<p>Also, could you close the other post as it is same question at the same time?<br>
<a href="https://discuss.pytorch.org/t/pytorch-using-1-of-gpu-the-training-is-slow-almost-like-on-cpu/86757">https://discuss.pytorch.org/t/pytorch-using-1-of-gpu-the-training-is-slow-almost-like-on-cpu/86757</a></p>

Jaredeco: <p>Hello, thank you very much for your help but when I try to send the optimizer to gpu(.cuda()) I get an error</p>
<pre><code class="lang-auto">AttributeError                            Traceback (most recent call last)
&lt;ipython-input-10-e7bad7cb8ff5&gt; in &lt;module&gt;
      1 import torch.optim as optim
      2 loss_fn = nn.CrossEntropyLoss()
----&gt; 3 opt = optim.Adam(net.parameters()).cuda()

AttributeError: 'Adam' object has no attribute 'cuda'
</code></pre>
<p><code>import torch.optim as optim loss_fn = nn.CrossEntropyLoss() opt = optim.Adam(net.parameters()).cuda()</code></p>

harsha_g: <p>Remove <code>.cuda()</code> after the <code>optim.Adam()</code> call. I did not know you could send the <code>optimizer</code> to any device. It’s usually only reserved for model or tensor. Happy to be told otherwise.</p>

Jaredeco: <p>Hello thank you for your suggestion, but I was told to do try to put optimizer to Gpu(opt.cuda()) because my code higher in this conversation is not almost using the Gpu. Do you have any suggestions please because I can not make this work for a really long time.</p>

Nikronic: <p>Ow, sorry, I do not know why I said <code>optim.cuda()</code> so strictly! thanks <a class="mention" href="/u/harsha_g">@harsha_g</a> for pointing out to. I meant loss functions but still it won’t make any differences because there is no parameter in cross entropy.<br>
Also, did you try to remove <code>Variable</code> from your code and sending in/out to GPU using <code>.cuda()</code>?</p>
<p>Sending model and input/output tensors to cuda is enough. Can you check that after sending input/output to cuda, they are really on cuda? <code>.device</code> will help.</p>
<p>See this thread, it has a lot of good responses to different issues. <a href="https://discuss.pytorch.org/t/solved-make-sure-that-pytorch-using-gpu-to-compute/4870" class="inline-onebox">[SOLVED] Make Sure That Pytorch Using GPU To Compute</a></p>

Nikronic: <p>Would you mind share a few examples of your data set and its structure? So, I can run few experiments.</p>

harsha_g: <aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://www.kaggle.com/static/images/favicon.ico" class="site-icon" width="32" height="32">
      <a href="https://www.kaggle.com/moltean/fruits" target="_blank" rel="nofollow noopener">kaggle.com</a>
  </header>
  <article class="onebox-body">
    <img src="https://storage.googleapis.com/kaggle-datasets-images/5857/8713/df2a86f4b6ecfff7996be179a4e8ebdf/dataset-card.jpg?t=2017-12-02-08-49-27" class="thumbnail onebox-avatar" width="500" height="500">

<h3><a href="https://www.kaggle.com/moltean/fruits" target="_blank" rel="nofollow noopener">Fruits 360</a></h3>

<p>A dataset with 90483 images of 131 fruits and vegetables</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>If I am not wrong <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

Jaredeco: <aside class="quote no-group" data-username="Nikronic" data-post="8" data-topic="86686">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/nikronic/48/71873_2.png" class="avatar"> Nikronic:</div>
<blockquote>
<p>See this thread, it h</p>
</blockquote>
</aside>
<p>i have tried everything removing Variable using .to(device) I read all the posts about this and none of them helped. But when I work with csv dataset and i do not use DataLoader it uses Gpu at about 60% like in this example</p>
<pre><code class="lang-auto">import torch 
import torch.nn as nn
import numpy as np
import pandas as pd
from torch.utils.data import DataLoader
from torch.autograd import Variable
import torch.nn.functional as F
from sklearn import preprocessing

nor = preprocessing.MinMaxScaler()
train_path = r"D:\Datasets\house prices\train.csv"
test_path = r"D:\Datasets\house prices\test.csv"
train_file = pd.read_csv(train_path)
test_file = pd.read_csv(test_path)
train_file.drop("Id", axis=1, inplace=True)
test_file.drop("Id", axis=1, inplace=True)
train_file.fillna(train_file.median(), inplace=True)
test_file.fillna(train_file.median(), inplace=True)
y_train = train_file["SalePrice"].values
y_train.resize(1460, 1)
num_cols = list(train_file._get_numeric_data().columns)
num_cols.remove("SalePrice")
train_file = pd.get_dummies(train_file, drop_first=True, dummy_na=True)
test_file = pd.get_dummies(test_file, drop_first=True, dummy_na=True)
train_file = nor.fit_transform(train_file)
test_file = nor.fit_transform(test_file)
X_train = train_file
x1 = pd.DataFrame(train_file)
x2 = pd.DataFrame(test_file)
x2 = x2.align(x1, axis=1)[0]
X_test = x2.values

X_train = Variable(torch.from_numpy(X_train).float())
y_train = Variable(torch.from_numpy(y_train).float())
X_test = Variable(torch.from_numpy(X_test).float())

class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        self.fc1 = nn.Linear(289, 128)
        self.fc2 = nn.Linear(128, 128)
        self.fc3 = nn.Linear(128, 128)
        self.fc4 = nn.Linear(128, 128)
        self.fc5 = nn.Linear(128, 1)
        self.drop = nn.Dropout(0.2)
    def forward(self, x):
        x = F.relu(self.fc1(x))
        x = self.drop(F.relu(self.fc2(x)))
        x = self.drop(F.relu(self.fc3(x)))
        x = F.relu(self.fc4(x))
        x = F.relu(self.fc5(x))
        return x
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
net = Model().cuda()

loss_fn = nn.MSELoss()
opt = torch.optim.Adam(net.parameters())

epochs = 10000
train_loss = 0
for epoch in range(epochs):
    preds = net(X_train.cuda())
    opt.zero_grad()
    loss = loss_fn(preds, y_train.cuda())
    loss.backward()
    opt.step()
    train_loss += loss.item()
    if epoch % 100 == 0:
        print(f"Loss: {loss.item()}")
</code></pre>
<p>this works with no problems, I think that there might be some problem with DataLoder when passing images to model. I really appreciate your help and time thank you so much</p>

harsha_g: <p>I just ran the code with the data and I find no reason to believe there’s anything wrong with the code. I ran it for 100 epochs with a batch size of 2500 and I can see a peak of 60% on my GPU (Tesla P100). Each epoch at most takes 5 seconds.</p>

Jaredeco: <aside class="quote no-group" data-username="harsha_g" data-post="12" data-topic="86686">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/h/4491bb/48.png" class="avatar"> harsha_g:</div>
<blockquote>
<p>takes</p>
</blockquote>
</aside>
<p>when i tried to run it with 2500 batch size, it did not use gpu at all the only thing that it used was ram. I have Gpu gtx1050 ti and Tensorflow runs no problem and I did not even get to 1 epoch in 5 mins. I do not what might be the problem this is like nightmare nothings works to solve that.</p>

harsha_g: <aside class="quote no-group" data-username="Jaredeco" data-post="13" data-topic="86686">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jaredeco/48/25963_2.png" class="avatar"> Jaredeco:</div>
<blockquote>
<p>I did not even get to 1 epoch in 5 mins.</p>
</blockquote>
</aside>
<p>Are you saying that even after 5 minutes, you did not finish at least 1 epoch?</p>

Jaredeco: <aside class="quote no-group" data-username="Jaredeco" data-post="13" data-topic="86686">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jaredeco/48/25963_2.png" class="avatar"> Jaredeco:</div>
<blockquote>
<p>not</p>
</blockquote>
</aside>
<p>yes it is so slow I do not know why I will try to run it in colab and see if the problem is in my pc.</p>

Jaredeco: <p>Now i tried it with Cifar10 dataset and it used gpu only sometimes and max 20% it was very  slow</p>

Jaredeco: <p>I reinstalled torch and now cifar10 from pytorch datasets uses gpu 50% but no improvements in that fruit dataset but i think that dataset is not a problem because tensorflow model worked fine with this fruit dataset.</p>

Jaredeco: <p>Please can anyone help with this I have been stuck making this cnn for a long time.</p>

Jaredeco: <p>I have just found out that there is no problem in model or cuda but there must be a problem in Dataset or DataLoader because Cifar10  from pytorch runs no problem. But I think that dataset is not a problem because there is a lot of models on GitHub with fruits datasets and it also does not work with dogs vs cats dataset so I think that there is some problem with custom datasets and daloader. Do you have any suggestions please. Thanks for  your help.</p>

harsha_g: <p><a class="mention" href="/u/jaredeco">@Jaredeco</a> Did you try playing with the <code>num_workers</code> argument <img src="https://discuss.pytorch.org/images/emoji/apple/point_right.png?v=9" title=":point_right:" class="emoji" alt=":point_right:"> <a href="https://discuss.pytorch.org/t/guidelines-for-assigning-num-workers-to-dataloader/813/19" class="inline-onebox">Guidelines for assigning num_workers to DataLoader</a></p>
<p>Other than that, I don’t really have much to add. I hope your issue is resolved quickly <img src="https://discuss.pytorch.org/images/emoji/apple/+1.png?v=9" title=":+1:" class="emoji" alt=":+1:">.</p>