CPU maxed out on training resnext50_32x4d....while gpu not being used hence slow training

username: mltp_soumya
cooked: <p>While training my model with pretrained resnext50_32x4d on virtual machine(kaggle virtual machine),the training is very slow and the cpu is being maxed while gpu remains unused. Cannot sort out or find the root of the problem…might be simple or basic one but me being new to pytorch and less experienced it’s bugging me.<br>
Dataset is from this kaggle competition <a href="https://www.kaggle.com/c/siim-isic-melanoma-classification/data?select=train.csv" rel="noopener nofollow ugc">link</a></p>
<blockquote>
<p><strong>Model</strong><br>
class Resnext50_32x4d(nn.Module):<br>
def <strong>init</strong>(self):<br>
super().<strong>init</strong>()<br>
self.model = models.resnext50_32x4d(pretrained = True)<br>
self.l1 = nn.Linear(1000,1)</p>
</blockquote>
<pre><code>def forward(self,image,view=True):
    #if view==True : print("Image shape {}".format(image.shape))
    img = self.model(image)
    out = self.l1(img)
    #print("Output Shape {}".format(out.shape))
    
    return out
</code></pre>
<blockquote>
<p><strong>Dataloader</strong><br>
class Data_Loader(Dataset):<br>
def <strong>init</strong>(self,image_path,im_name,target,valid=False):<br>
self.name = im_name<br>
self.target = target<br>
self.path = image_path<br>
self.valid = valid</p>
</blockquote>
<pre><code>def __len__(self):
    return (len(self.name))

def __getitem__(self,index):
    
    if self.valid==False:
        im = self.name[index]
        self.train_y = self.target[index]
        im_tensor = image_aug(self.path,im)
        
        return im_tensor,self.train_y
    
    else:
        im = self.name[index]
        self.valid_y = self.target[index]
        im_tensor = image_aug(self.path,im,valid=True)
        
        return im_tensor,self.valid_y
</code></pre>
<blockquote>
<p><strong>Training</strong><br>
batch_t = 32<br>
batch_v = 16<br>
device = ‘cuda’<br>
image_path = ‘/kaggle/input/siim-isic-melanoma-classification/jpeg/train/’<br>
train_df = df[df.kfold!=0].reset_index(drop=True)<br>
valid_df = df[df.kfold==0].reset_index(drop=True)<br>
train_im = train_df.image_name.values.tolist()<br>
train_y = train_df.target.values<br>
valid_im = valid_df.image_name.values.tolist()<br>
valid_y = valid_df.target.values<br>
train_dataset = Data_Loader(image_path,train_im,train_y)<br>
train_dataset = DataLoader(train_dataset,batch_t,shuffle=False,num_workers=4)<br>
valid_dataset = Data_Loader(image_path,valid_im,valid_y)<br>
valid_dataset = DataLoader(valid_dataset,batch_v,shuffle=False,num_workers=4)</p>
</blockquote>
<blockquote>
<p><strong>train loop</strong><br>
def train(fold):<br>
model = Resnext50_32x4d()<br>
model = model.cuda()<br>
optimizer = torch.optim.Adam(model.parameters(), lr=1e-4)<br>
scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(<br>
optimizer,<br>
patience=3,<br>
threshold=0.001,<br>
mode=“max”<br>
)<br>
criterion = nn.BCEWithLogitsLoss()<br>
epochs = 25<br>
best_score = 0</p>
</blockquote>
<pre><code>for epoch in range(epochs):
    
        model.train()
        batch = 0 
        for train_data,label in train_dataset:
            train_data = train_data.to(device)
            label = torch.tensor(label,dtype = torch.float32)
            label = label.to(device)
            
            optimizer.zero_grad()
            out = model(train_data)
            loss = criterion(out,label.unsqueeze(1).type_as(out))
            batch +=1
            #print("EPOCH {}  Loss {}  batch  {}".format(epoch,loss.item(),batch))
            
            loss.backward()
            optimizer.step()
        

        model.eval()
        true = []
        pred = []
        batch = 0
        for valid_data,valid_label in valid_dataset:
            valid_data = valid_data.to(device)
            valid_label = torch.tensor(valid_label,dtype = torch.float32)
            valid_label = valid_label.to(device)
            batch +=1
            true.append(valid_label.cpu())
            
            with torch.no_grad():
                out = model(valid_data)
                loss = criterion(out,valid_label.unsqueeze(1).type_as(out))
                pred.append(out.cpu())
               # print('Valid Loss {}  batch  {}'.format(loss.item(),batch))
        #print("true   {} '''/n''' pred  {}".format(true,pred))     
        #true=np.vstack((true)).ravel()#torch.tensor(true).view(-1)
        #pred=np.vstack((pred)).ravel()#torch.tensor(pred).view(-1)
        #auc_score = roc_auc_score(true,pred)
        #print("EPOCH {}  AUC Score {}".format(epoch,auc_score))
                
        #if auc_score&gt;best_score:
                    #best_score = auc_score 
                    #torch.save(model,'best_model.pth')
                    #print("Validation Score Improved ======&gt;&gt;&gt;&gt;&gt;&gt; Saving Model")
</code></pre>

username: ptrblck
cooked: <p>You could try to narrow down the bottleneck by profiling the code.<br>
E.g. to isolate a data loading bottleneck, you could use the <code>data_time</code> class from the <a href="https://github.com/pytorch/examples/blob/59caa16a29476f290d2e4d38162a89372e5ccb15/imagenet/main.py#L280">ImageNet example</a>. If the data loading is the bottleneck, you could have a look at <a href="https://discuss.pytorch.org/t/how-to-prefetch-data-when-processing-with-gpu/548/19">this post</a> for potential reasons of this bottleneck and workarounds. Of course you won’t be able to change the machine, as it’s a Kaggle node.</p>