Slow GPU execution Pytorch 1.13.1

deep_dabbler: <p>My system is as follows:</p>
<p>NVIDIA Titan V<br>
AMD Threadripper 32-core CPU with 64MB RAM<br>
running Pytorch 1.13.1 + cuda 11.6  installed on Anaconda Python 3.9.x<br>
I’m using Ubuntu 18.04 with kernel 5.14 and nvidia-driver version 525.78.01</p>
<p>My code was adapted from the Pytorch finetuning example code. I’m training a image classification model based on torchvision’s MobileNetV3 (Small) architecture. I’m only retraining the model.classifier layers.</p>
<p>Its extremely frustrating to see Pytorch takes 2.5 sec to train a single batch. Pytorch reports device <code>cuda:0</code> is available, but my GPU activity seems very low or inactive when training the model (observed using <code>nvidia-smi -l</code>).<br>
My code is as follows:</p>
<pre><code class="lang-auto">import torch
from utils.dataloader import MyCustomDataset
import copy

from sklearn.model_selection import KFold
import sklearn.preprocessing
from torch import nn
import torch.optim as optim
from torch.optim import lr_scheduler
from torchvision import models,utils
import torchvision.transforms as T
#import matplotlib.pyplot as plt
from torch.utils.tensorboard import SummaryWriter
from torchmetrics.classification import BinaryAccuracy
from tqdm import tqdm




def train_model(model, criterion, optimizer, scheduler, batch_size, num_epochs=10): 

    best_model_wts = copy.deepcopy(model.state_dict())
    best_acc = 0.0

    kfold = KFold(n_splits=3)
    
    
    for fold, (train_index, val_index) in enumerate(kfold.split(deepstage_dataset)):
                # Print
        print(f'FOLD {fold}')
        print('=' * 6)
        
        
        dataloaders={}     
        dataloaders['train'] = torch.utils.data.DataLoader(deepstage_dataset, batch_size = batch_size, shuffle = True,num_workers=16)
        dataloaders['val'] = torch.utils.data.DataLoader(deepstage_dataset, batch_size = batch_size, shuffle = True,num_workers=16)
        
        dataset_sizes = {x: len(dataloaders[x]) for x in ['train', 'val']}



        for epoch in range(num_epochs):
            print()
            
            # Each epoch has a training and validation phase
            for phase in ['train', 'val']:
                if phase == 'train':
                    model.train()  # Set model to training mode
                else:
                    model.eval()   # Set model to evaluate mode

                running_loss = 0.0
                running_accuracy = 0.0


                loop = tqdm(enumerate(dataloaders[phase]),total=len(dataloaders[phase]),leave=False)
                # Iterate over data.
                for batch_idx,sample in loop:
                    inputs = sample['image'].to(device=torch.device('cuda'),dtype=torch.float32)
                    labels = torch.nn.functional.one_hot(sample['label'],num_classes).to(device=torch.device('cuda'), dtype=torch.float32)
      
              

                    grid = utils.make_grid(inputs)
                    writer.add_image('images', grid, epoch)

                    # zero the parameter gradients
                    optimizer.zero_grad()

                    # forward
                    # track history if only in train
                    with torch.set_grad_enabled(phase == 'train'):
                        output = model(inputs)
                        loss = criterion(output, labels)
                        writer.add_scalar("Loss[train]", loss, epoch)
                        

                        # backward + optimize only if in training phase
                        if phase == 'train':
                            loss.backward()
                            optimizer.step()
                    

                    # statistics
                
                    #accuracy = Accuracy(task="multiclass", num_classes=num_classes).to(device)
                    accuracy = BinaryAccuracy().to(device=torch.device('cuda'))
                    running_accuracy += accuracy(output, labels).cpu().numpy()
                    running_loss += loss.item() * inputs.size(0)

                    
                    
                    #update progress bar
                    loop.set_description(f"Epoch [{epoch+1}/{num_epochs}]")
                    loop.set_postfix(loss=running_loss, acc=running_accuracy)
                    
                if phase == 'train':
                    scheduler.step()

                epoch_loss = running_loss / dataset_sizes[phase]
                epoch_acc = (running_accuracy / dataset_sizes[phase])
            
                writer.add_scalar("Accuracy[train]", epoch_acc, epoch)

                
                if phase == 'val':
                    writer.add_scalar("Accuracy[val]", epoch_acc, epoch)

                
                print(f'{phase} loss: {epoch_loss:.4f} acc: {epoch_acc:.4f} ')

                # deep copy the model
                if phase == 'val' and epoch_acc &gt; best_acc:
                    best_acc = epoch_acc
                    best_model_wts = copy.deepcopy(model.state_dict())


        # load best model weights
        model.load_state_dict(best_model_wts)
    
        
        return model





if __name__ == "__main__":

    # Training parameters
    data_file = '../../datasets/resized_dataset.csv'
    num_classes = 6
    feature_extract=True
    batch_size = 64
    writer = SummaryWriter()
    
    torch.backends.cudnn.benchmark = True

    preprocess = T.Compose([
    T.ToPILImage(),
    T.CenterCrop(224),
    T.ToTensor(),
    T.Normalize(
        mean=[0.485, 0.456, 0.406],
        std=[0.229, 0.224, 0.225]
    )
    ])

    device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
    dataset = MyCustomDataset(data_file=data_file,transform = preprocess)
   
   

    model_ft = models.mobilenet_v3_small(weights='MobileNet_V3_Small_Weights.DEFAULT') 
    print(model_ft)
    model_ft.classifier[3] = nn.Linear(in_features=1024, out_features=num_classes, bias=True)
    
    
    
    def set_parameter_requires_grad(model, feature_extracting):
        if feature_extracting:
            for param in model.parameters():
                param.requires_grad = False
    
    set_parameter_requires_grad(model_ft.features, feature_extract)

    
   
    
    # Gather the parameters to be optimized/updated in this run. If we are
    #  finetuning we will be updating all parameters. However, if we are
    #  doing feature extract method, we will only update the parameters
    #  that we have just initialized, i.e. the parameters with requires_grad
    #  is True.
    params_to_update = model_ft.parameters()
    print("Params to learn:")
    if feature_extract:
        params_to_update = []
        for name,param in model_ft.named_parameters():
            if param.requires_grad == True:
                params_to_update.append(param)
                print("\t",name)
    else:
        for name,param in model_ft.named_parameters():
            if param.requires_grad == True:
                print("\t",name)

    
   
    criterion = nn.CrossEntropyLoss()

    #per layer learning rate
    #optimizer_ft = optim.Adam(params_to_update, lr= 1e-2)
    optimizer_ft = optim.Adam([
                {'params': model_ft.features.parameters()},
                {'params': model_ft.classifier.parameters(), 'lr': 1e-2}
            ], lr=1e-4)

    # Decay LR by a factor of 0.1 every 7 epochs
    exp_lr_scheduler = lr_scheduler.StepLR(optimizer_ft, step_size=7, gamma=0.1)
    model_ft = model_ft.to(device)
    model_ft = train_model(model_ft, criterion, optimizer_ft, exp_lr_scheduler,
                        batch_size=batch_size,num_epochs=35)
    
    torch.save(model_ft,'saved_weights/savedmodel.pth')
    writer.flush()
    writer.close()


</code></pre>
<p>since Pytorch conda installtion ships with its own cuda and cudnn libraries,  would having a driver that uses CUDA 12.0 (as shown in nvidia-smi`) be the culprit</p>

deep_dabbler: <p>FYI, the dataloader class is:</p>
<pre><code class="lang-auto">import os
import torch
import numpy as np
import pandas as pd
from torch.utils.data import Dataset,DataLoader
import torchvision.transforms as T
from PIL import Image





class MyCustomDataset(Dataset):
    
    def __init__(self, data_file, transform=None, target_transform=None):
        self.data_file = data_file
        self.img_list, self.lbl_list = self._get_data(self.data_file)
        self.transform = transform
        self.target_transform = target_transform
       
        
        
    def __len__(self):
        return len(self.img_list)
    
    def __getitem__(self,idx):
        if torch.is_tensor(idx):
            idx = idx.tolist()

        
        img_path = self.img_list[idx]
        lbl = self.lbl_list[idx]
        
        image = np.asarray(Image.open(img_path))
        label = np.asarray(lbl)
      
        
        if self.transform:
           transformed = self.transform(image)
           image = transformed
        if self.target_transform:
            label = self.target_transform(label)
        
        sample = {'image': image, 'label': label}
        return sample
    
    def _get_data(self,input):
        with open(input) as f:
            df = pd.read_csv(f)
        imagelist = df['FILEPATH'].to_list()
        labellist = df['LABEL'].to_list()
        return imagelist,labellist
        
        
    

</code></pre>

ptrblck: <p>No, the driver should not cause any slowdowns and I would recommend profiling the actual workload to see where the bottleneck is coming from.<br>
You could use the native PyTorch profiler or e.g. Nsight Systems as described <a href="https://dev-discuss.pytorch.org/t/using-nsight-systems-to-profile-gpu-workload/59">here</a>.<br>
As a quick test you could also profile the <code>DataLoader</code> alone and see how long it takes to load and process a single batch.</p>

tiramisuNcustard: <p><a class="mention" href="/u/deep_dabbler">@deep_dabbler</a>, Just curious - is 64 MB your RAM size or is it one of the CPU cache sizes?</p>

deep_dabbler: <p><a class="mention" href="/u/tiramisuncustard">@tiramisuNcustard</a> Sorry, it has 64GB(!) of RAM. Thanks for pointing it out.</p>
<p><a class="mention" href="/u/ptrblck">@ptrblck</a> ,  I ran the tensorboard profiler and I’m seeing extremely low GPU utilization. I changed the batch size from 64 to 256 - this made no difference to the GPU utilization at all.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/c/4c6dd4a80b1e832537eae8b9e64e6914240d2045.png" data-download-href="https://discuss.pytorch.org/uploads/default/4c6dd4a80b1e832537eae8b9e64e6914240d2045" title="Screenshot 2023-01-30 at 7.53.55 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/c/4c6dd4a80b1e832537eae8b9e64e6914240d2045_2_690x459.png" alt="Screenshot 2023-01-30 at 7.53.55 PM" data-base62-sha1="aU7B8fuyDv5mUJnKJbM7KEoTn4F" width="690" height="459" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/c/4c6dd4a80b1e832537eae8b9e64e6914240d2045_2_690x459.png, https://discuss.pytorch.org/uploads/default/optimized/3X/4/c/4c6dd4a80b1e832537eae8b9e64e6914240d2045_2_1035x688.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/4/c/4c6dd4a80b1e832537eae8b9e64e6914240d2045_2_1380x918.png 2x" data-dominant-color="F2D6D9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2023-01-30 at 7.53.55 PM</span><span class="informations">2557×1702 319 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I also profiled the dataloader, that seems perfectly fine.<br>
I’m actually reading a batch of images and and int label from a list in the dataloader and converting the numpy array (the batch of labels/targets) into a one-hot encoding within the train function. I’ve also converted both image batch and label tensors to CUDA tensors.<br>
I’ve ran this code om two different workstations with same results. <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=12" title=":frowning:" class="emoji" alt=":frowning:" loading="lazy" width="20" height="20"></p>

ptrblck: <p>I don’t know what “Other” refers to in your output. Do you see any more information about this part of the profile?<br>
If not, could you create a new profile using Nsight Systems and check the timeline, please?</p>

deep_dabbler: <p>I think I finally figured out the bottleneck! In my training loop, I had set up a tensorboard visualization of the input images after post-processing for sanity check.</p>
<pre><code class="lang-auto">                grid = utils.make_grid(inputs)
                writer.add_image('images', grid, epoch)
</code></pre>
<p>I was under the impression that this function only ran at the beginning of each epoch, but it seems that it was perhaps running on each batch, and slowing down the training significantly. Took me a few days to figure this out…but glad I did. Thanks for all the help!</p>

ptrblck: <p>Good to hear you’ve narrowed down the bottleneck!</p>