GPU not being used

Wafaa_Wardah: <p>I’m trying to run my CNN training and testing on GPU but it’s not using GPU <img src="https://discuss.pytorch.org/images/emoji/apple/pensive.png?v=9" title=":pensive:" class="emoji" alt=":pensive:"></p>
<p>I am stating</p>
<p><code>model = models.CNN1().cuda()</code><br>
<code>inputs = Variable(inputs).cuda()</code><br>
<code>labels = Variable(labels).cuda()</code><br>
<code>outputs = model(inputs)</code></p>
<p>And when I print out device, it does say  <code>Device: cuda:0</code></p>
<p>Please help. <img src="https://discuss.pytorch.org/images/emoji/apple/pensive.png?v=9" title=":pensive:" class="emoji" alt=":pensive:"></p>

ptrblck: <p>It looks like the GPU0 is being used.<br>
What does <code>nvidia-smi</code> show? Any memory allocation and utilization?</p>

Wafaa_Wardah: <p><img src="https://discuss.pytorch.org/uploads/default/original/2X/2/22bea2718a05d6c0591c9bb7bbdb22226b951dd4.png" alt="image" data-base62-sha1="4XmFkvd5wZwRip9FXys6lBa9c2M" width="636" height="160"></p>

ptrblck: <p>Do you see the utilization and memory increase at some point or is it the whole time at 0%?<br>
Also, what kind of model are you using? Is it quite small so that the workload might be minimal compared to the data loading, processing, etc.?</p>

Wafaa_Wardah: <p>It does change to 3%, 5% sometimes, then back to 0%.<br>
I have various cnn models (none producing great results yet). Shallow and deep, none use GPU like I want it to.</p>
<p>example shallow model:</p>
<pre><code class="lang-auto">class CNN1(nn.Module):
    def __init__(self):
        super(CNN1, self).__init__()
        self.layer1 = nn.Sequential(nn.Conv2d(1, 10, 5, 1),
                                   nn.ReLU(),
                                   nn.MaxPool2d(2, 2))
        self.fc1 = nn.Linear(170, 2)
        
    def forward(self, x):
        x = x.unsqueeze(1)
        x = self.layer1(x)
        x = x.reshape(x.size(0), -1)
        x = self.fc1(x)
        return x
</code></pre>
<p>example deeper model:</p>
<pre><code class="lang-auto">class CNN2(nn.Module):
    def __init__(self):
        super(CNN2, self).__init__()  
        self.layer1 = nn.Sequential(
            nn.Conv2d(1, 32, kernel_size=5, stride=1, padding=2),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2, stride=2))
        self.layer2 = nn.Sequential(
            nn.Conv2d(32, 64, kernel_size=5, stride=1, padding=2),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2, stride=2))
        self.drop_out = nn.Dropout(0.5)
        self.fc1 = nn.Linear(576, 1000)
        self.fc2 = nn.Linear(1000, 1000)
        self.fc3 = nn.Linear(1000, 100)
        self.fc4 = nn.Linear(100, 2)
        
    def forward(self, x):
        x = x.unsqueeze(1)
        out = self.layer1(x)
        out = self.layer2(out)
        out = out.reshape(out.size(0), -1)
        out = out.float()
        out = self.drop_out(out)
        out = self.fc1(out)
        out = self.fc2(out)
        out = self.fc3(out)
        out = self.fc4(out)
        return out
</code></pre>
<p>The CNN2 model sometimes shows 14%, 24%, then 0% mostly. But Task Manager performance still shows 0% for GPU <img src="https://discuss.pytorch.org/images/emoji/apple/pensive.png?v=9" title=":pensive:" class="emoji" alt=":pensive:"></p>
<p>I am loading data by loading files in <strong>getitem</strong>() … Is this causing it?</p>

ptrblck: <p>Loading the data in the <code>Dataset</code>'s <code>__getitem__</code> is a good idea.<br>
Are you using multiple workers in your <code>DataLoader</code> (<code>num_workers &gt; 0</code>)?<br>
Is your data stored on a local SSD or are you pulling it from a network device?</p>
<p>PS: I would recommend to use <code>tensor.view</code> instead of <code>tensor.reshape</code> in your <code>forward</code> method. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

Wafaa_Wardah: <p>Thanks <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>I tried using  <code>num_workers &gt; 0</code> but no difference. I even set batch size to 1024. CPU shows 100% and GPU 0%.<br>
My data is stored locally.</p>

ptrblck: <p>Are you applying a lot of preprocessing?<br>
If you are using <code>PIL</code>, you might use <code>Pillow-SIMD</code>, which is a drop-in replacement for Pillow with SIMD.<br>
Could you post the <code>__getitem__</code> method so that we can have a look at potentially speedups?</p>

Wafaa_Wardah: <pre><code class="lang-auto">    def __getitem__(self, index):
        with open(self.folder + str(index) + '.txt', 'r') as file:
            buffer = file.read()
            buffer = buffer.split('\n')
            label = int(buffer.pop(0).split('\t')[1])
            buffer.pop()
            image = []
            for line in buffer:
                line = line.split('\t')[1:-1]
                line = [float(x) for x in line]
                image.append(line)
        return torch.tensor(image), torch.tensor(label)
</code></pre>

ptrblck: <p>Oh, your images are stored in text files?<br>
If that’s the case, I could suggest to load them all once and store them in an image format or binary format using <code>torch.save</code>.<br>
This should speed thing up.</p>

Wafaa_Wardah: <p>Yes I saved them in text files. Initially I saved them as pickle objects in one file and loaded all in a list in the <code>__init__()</code> but that filled up my RAM and all crashed, so I then saved the images in separate text files.</p>
<p>I’m not sure how you want me to use torch.save for saving image files. I thought that was for saving the model? Sorry, please could you explain? Thanks.</p>

ptrblck: <p><code>torch.save</code> can also save tensors, not only the <code>state_dict</code>.<br>
You could add the following line to your current <code>__getitem__</code> and store each image tensor in a folder corresponding to the label:</p>
<pre><code class="lang-python">...
for line in buffer:
   ...
x = torch.tensor(image)
torch.save(x, self.folder + '/' + str(label) + '/' + str(index) + '.pth')
return x, torch.tensor(label)
</code></pre>
<p>This will save each image tensor in a folder given by <code>str(label)</code>.<br>
The file names won’t be ascending, i.e. you might end up with:</p>
<pre><code class="lang-auto">root/
── 0/
│   ├── 24.pth
│   ├── 4678.pth
│   ├── 4679.pth
│   └── ...
├── 1/
│   ├── 0.pth
│   └── 1.pth
</code></pre>
<p>After saving all files (just run this code for one single epoch), you should rewrite your <code>Dataset</code> by providing the file paths to its <code>__init__</code> and just loading each image tensor using <code>torch.load</code> in <code>__getitem__</code>.<br>
Let me know if you get stuck somewhere.</p>
<p>PS: Alternatively, you could also save the images using <code>PIL</code> and later just use an <code>ImageFolder</code> as your dataset.</p>

Wafaa_Wardah: <p>I created the files using torch.save and fixed my Dataset code. But still getting 0% GPU usage <img src="https://discuss.pytorch.org/images/emoji/apple/pensive.png?v=9" title=":pensive:" class="emoji" alt=":pensive:"> Please help.</p>

ptrblck: <p>Could you add the data loading profiling into your code as seed in the <a href="https://github.com/pytorch/examples/blob/5df464c46cf321ed1cc3df1e670358d7f5ae1887/imagenet/main.py#L274" rel="nofollow noopener">ImageNet example</a>? If the time doesn’t decrease towards zero, your data loading is still a bottleneck.</p>

Wafaa_Wardah: <p>Okay I tried that and found that time is not decreasing, or is it?</p>
<pre><code class="lang-auto">Epoch: [0][0/3]         Time  2.614 ( 2.614)	
Epoch: [1][0/3]	        Time  0.595 ( 0.595)	
Epoch: [2][0/3]	        Time  0.592 ( 0.592)	
...                          
Epoch: [88][0/3]        Time  0.586 ( 0.586)
Epoch: [89][0/3]        Time  0.596 ( 0.596)
</code></pre>
<p>What do you think I should do? <img src="https://discuss.pytorch.org/images/emoji/apple/pensive.png?v=9" title=":pensive:" class="emoji" alt=":pensive:"></p>

ptrblck: <p>What kind of drive are you using? Is The data on a SSD or HDD?</p>

Wafaa_Wardah: <p>My data is on HDD <img src="https://discuss.pytorch.org/images/emoji/apple/worried.png?v=9" title=":worried:" class="emoji" alt=":worried:"></p>

ptrblck: <p>That explains the long loading time.<br>
Do you have a (small) SSD where you could put the data on? If not, you can’t really speed up the loading process, since your magnetic drive is the bottleneck here.<br>
Loading all the data into RAM would speed up the process after an initial loading time, but as you said the dataset is too large to fit into memory so you are stuck with lazy loading.</p>

Wafaa_Wardah: <p>Okay I’ll try to get my hands on SSD. In case I can’t and need to load all data into the RAM, do you think I can somehow split my data and somehow use a load-train-load-train—validate–test kind of strategy?</p>

ptrblck: <p>This should be possible and might be seen as some kind of prefetching. However, I think a SATA SSD (or NVME SSD drive if possible) would yield a significant and easy performance boost.</p>