Low GPU (GTX 1070) utilization while training FCN

WeeWooWeeWoo: <p>Hello,</p>
<p>I am currently trying to train an FCN on my GPU. When I train the FCN, I see that the memory usage of my GPU is very high, but the processing utilization is very low. Is this normal? If not, could there be something that I am doing wrong?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/0/001fac02fa2576618f42e92b325ece8b4c86ba7b.png" data-download-href="https://discuss.pytorch.org/uploads/default/001fac02fa2576618f42e92b325ece8b4c86ba7b" title="Picture of GPU utilization"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/0/001fac02fa2576618f42e92b325ece8b4c86ba7b_2_690x260.png" alt="Picture of GPU utilization" data-base62-sha1="15R7rQnJnETm70HqYvFq9nV1Rp" width="690" height="260" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/0/001fac02fa2576618f42e92b325ece8b4c86ba7b_2_690x260.png, https://discuss.pytorch.org/uploads/default/optimized/3X/0/0/001fac02fa2576618f42e92b325ece8b4c86ba7b_2_1035x390.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/0/0/001fac02fa2576618f42e92b325ece8b4c86ba7b_2_1380x520.png 2x" data-dominant-color="BDBFC0"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Picture of GPU utilization</span><span class="informations">1449×546 60.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p><strong>Some code snippets that may help</strong></p>
<pre data-code-wrap="python"><code class="lang-python"># dataset snippet
def make_dataset(segmentation_type, mode):
   if segmentation_type == 'semantic':
       mask_dir_name = 'labels'
   else:
       raise ValueError('Segmentation type {} unsupported'.format(segmentation_type))

   img_path = os.path.join(root, mode, 'images')
   mask_path = os.path.join(root, mode, mask_dir_name)
   
   assert len(os.listdir(img_path)) == len(os.listdir(mask_path))
   dataset = []
   names = [img.split('.jpg')[0] for img in os.listdir(img_path)]
   for name in names:
       data_item = (os.path.join(img_path, name + '.jpg'), os.path.join(mask_path, name + '.png'))
       dataset.append(data_item)
   return dataset

class Mapillary(Dataset):
   def __init__(self, segmentation_type, mode, joint_transform=None,
                sliding_crop=None, transform=None, target_transform=None):
                
       assert (segmentation_type in set(('panoptic', 'semantic', 'instance')))
       assert (mode in set(('training', 'validation')))
      
       self.imgs = make_dataset(segmentation_type, mode)
       self.mode = mode
       self.joint_transform = joint_transform
       self.sliding_crop = sliding_crop
       self.transform = transform
       self.target_transform = target_transform

   def __len__(self):
       return len(self.imgs)

   def __getitem__(self, idx):
       img_path, mask_path = self.imgs[idx]
       img, mask = Image.open(img_path).convert('RGB'), Image.open(mask_path)
       mask = np.array(mask)
       mask_copy = mask.copy()
       mask = Image.fromarray(mask_copy.astype(np.uint8))  

       if self.joint_transform is not None:
           img, mask = self.joint_transform(img, mask)

       if self.sliding_crop is not None:
           img_slices, mask_slices, slices_info = self.sliding_crop(img, mask)
           if self.transform is not None:
               img_slices = [self.transform(e) for e in img_slices]

           if self.target_transform is not None:
               mask_slices = [self.target_transform(e) for e in mask_slices]

           img, mask = torch.stack(img_slices, 0), torch.stack(mask_slices, 0)
           return img, mask, torch.LongTensor(slices_info)

       else:
           if self.transform is not None
               img = self.transform(img)

           if self.target_transform is not None:
               mask = self.target_transform(mask)

           return img, mask
       return img, mask 
</code></pre>
<pre data-code-wrap="python"><code class="lang-python"># dataloader
train_loader = DataLoader(train_set, batch_size=args['train_batch_size'], num_workers=2, shuffle=True)
</code></pre>
<pre data-code-wrap="python"><code class="lang-python"># train snippet
def train(train_loader, net, criterion, optimizer, epoch, train_args):
    train_loss = AverageMeter()
    curr_iter = (epoch - 1) * len(train_loader)
    for i, data in enumerate(train_loader):
        inputs, labels = data
        assert inputs.size()[2:] == labels.size()[1:]
        N = inputs.size(0)
        inputs = Variable(inputs).cuda()
        labels = Variable(labels).cuda()

        optimizer.zero_grad()
        outputs = net(inputs)
        assert outputs.size()[2:] == labels.size()[1:]
        assert outputs.size()[1] == mapillary.num_classes

        loss = criterion(outputs, labels) / N
        loss.backward()
        optimizer.step()

        train_loss.update(loss.data, N)

        curr_iter += 1
        writer.add_scalar('train_loss', train_loss.avg, curr_iter)

        if (i + 1) % train_args['print_freq'] == 0:
            print('[epoch %d], [iter %d / %d], [train loss %.5f]' % (
                epoch, i + 1, len(train_loader), train_loss.avg))
</code></pre>
<p>I saw online that increasing the num_workers attribute of the dataloader may increase GPU utilization, but I subsequently run into an out of memory error if I increase num_workers. Most of the code I am using was adapted from <a href="https://github.com/zijundeng/pytorch-semantic-segmentation" class="inline-onebox" rel="noopener nofollow ugc">GitHub - zijundeng/pytorch-semantic-segmentation: PyTorch for Semantic Segmentation</a>. I am quite new to all of this, so any advice would be greatly appreciated.</p>
<p>Thanks!</p>

Pmimo21: <p>Hello,</p>
<p>For me it’s because your GPU is not doing 3D, Video Encoding / Decoding, is doing Cuda stuff.<br>
Select the cuda tab instead of Video Encode / Decode to see some stuff.</p>