Pytorch taking hours of time train a single batch

username: Sheethal00
cooked: <p>Hi i have neural network to classify an image as eye or not.</p>
<p>I am training the neural network with batch size 32, for some iterations say about 70 the model is giving predictions faster but 98th iteration started at 00:48:18, still this iteration is finished.</p>
<p>Could anyone please tell me why the network is taking so much time.</p>
<p>Thanks in advance.</p>

username: Sheethal00
cooked: <p>The 98th iteration is still not finished.</p>
<p>Here is my network architecture.</p>
<p>class Net(torch.nn.Module):<br>
def <strong>init</strong>(self):<br>
super(Net, self).<strong>init</strong>()<br>
self.conv1 = nn.Conv2d(3, 32, 3)<br>
torch.nn.init.xavier_normal_(self.conv1.weight)<br>
self.pool1 = nn.MaxPool2d(3,stride=2,ceil_mode=True)<br>
self.conv2 = nn.Conv2d(32, 64, 3)<br>
torch.nn.init.xavier_normal_(self.conv2.weight)<br>
self.pool2 = nn.MaxPool2d(3,stride=2,ceil_mode=True)<br>
self.conv3 = nn.Conv2d(64, 64, 3)<br>
torch.nn.init.xavier_normal_(self.conv3.weight)<br>
self.pool3 = nn.MaxPool2d(2,stride=2,ceil_mode=True)<br>
self.conv4 = nn.Conv2d(64, 128, 2)<br>
torch.nn.init.xavier_normal_(self.conv4.weight)<br>
self.fc1 = nn.Linear(3<em>1</em>128, 256)<br>
self.cls_prob_fc = nn.Linear(256,2)<br>
self.bbox_pred_fc = nn.Linear(256,4)<br>
self.lm_pred_fc = nn.Linear(256,12)</p>
<pre><code>def forward(self, x):
    x = self.pool1(F.relu(self.conv1(x)))
    x = self.pool2(F.relu(self.conv2(x)))
    x = self.pool3(F.relu(self.conv3(x)))
    x = F.relu(self.conv4(x))
    x = x.view(x.size(0), 1*3*128)
    x = F.relu(self.fc1(x))
    x_cls_prob = self.cls_prob_fc(x)
    x_bbox_pred = self.bbox_pred_fc(x)
    x_lm_pred = self.lm_pred_fc(x)    
    return x_cls_prob,x_bbox_pred,x_lm_pred</code></pre>

username: Kushaj
cooked: <p>Show code for the training loop also. The model seems ok.</p>

username: Sheethal00
cooked: <p>def train(self):<br>
# print(“Training the batch…”)<br>
# Ground truth<br>
indexes = torch.tensor([self.samples.index(obj) for obj in self.samples if obj.dtype==“pos”])<br>
gt_imgs = [obj.patch for obj in self.samples if obj.dtype==“pos”]<br>
cls_label = torch.tensor([1 if obj.dtype==“pos” else 0 for obj in self.samples])<br>
bbox_gt = torch.FloatTensor([self.PosSet.pos_samples[obj.id].patch_bbox for obj in self.samples if obj.dtype==“pos”])<br>
bbox_gt = torch.reshape(bbox_gt,(-1,BBOX))  # TODO: BS // convert 4 to derived macro; or compute from existing macros<br>
lm_gt = torch.FloatTensor([self.PosSet.pos_samples[obj.id].patch_lm.flatten() for obj in self.samples if obj.dtype==“pos”])<br>
lm_gt = torch.reshape(lm_gt,(-1,LM))  # TODO: BS // Why 12? Derive using existing macros and variables.</p>
<pre><code>    # Training
    image_batch = torch.FloatTensor([obj.patch for obj in self.samples])
    image_batch = image_batch.reshape(BATCH_SIZE,3,TEMPLATE_HGT,TEMPLATE_WDT)    
    pred = model(image_batch)
    optimizer.zero_grad()
    
    # Predictions
    cls_pred = pred[0]
    bbox_pred = torch.index_select(pred[1],0,indexes)
    lm_pred = torch.index_select(pred[2],0,indexes)
    
    # Loss
    cls_loss = criterion_cls(cls_pred,cls_label)
    bbox_loss = criterion_bbox(bbox_pred,bbox_gt)
    bbox_loss = torch.mean(bbox_loss,1)
    lm_loss = criterion_lm(lm_pred,lm_gt)
    lm_loss = torch.mean(lm_loss,1)
    
    # Sum up the loss
    nsamples = len([obj for obj in self.samples if obj.dtype=="neg"])
    dummy_loss = torch.zeros(nsamples)
    bbox_loss = torch.cat((bbox_loss,dummy_loss))
    lm_loss = torch.cat((lm_loss,dummy_loss))
    loss = A * cls_loss + B * bbox_loss + (1 - A - B) * lm_loss
    loss.sum().backward()
    optimizer.step()
    return loss</code></pre>

username: Kushaj
cooked: <p><code>return loss</code> I think this is the problem. Can you try with <code>return loss.detach()</code>.</p>
<p>Problems like these are generally caused when you training loop is holding on to some things that it shouldn’t. Also, make sure that you are not storing some temporary computations in an ever growing list without deleting them.</p>
<p>To track this down, you can also get timings for different parts separately: data loading, network forward, loss computation, backward pass and parameter update. Hopefully just one will increase and you will be able to see better what is going on.</p>