Pytorch question : loss backward takes too much time!

lifeblack: <p>Hello!<br>
I have a question about simulation of  a network by Pytorch.<br>
This is the network that I  want o simulate:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/3/23dd761dd3c80962a8de5d1b043aa4583376eaa6.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/23dd761dd3c80962a8de5d1b043aa4583376eaa6" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/3/23dd761dd3c80962a8de5d1b043aa4583376eaa6_2_690x351.jpeg" alt="image" data-base62-sha1="57hbNytuk7JlUaq9QRjpXqdxId8" width="690" height="351" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/3/23dd761dd3c80962a8de5d1b043aa4583376eaa6_2_690x351.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/2/3/23dd761dd3c80962a8de5d1b043aa4583376eaa6.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/2/3/23dd761dd3c80962a8de5d1b043aa4583376eaa6.jpeg 2x" data-dominant-color="E9E9E9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">853×435 72.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
and I wrote it’s code:</p>
<pre><code class="lang-auto">class ResidualLearningNet(nn.Module):
    def __init__(self):
        super(ResidualLearningNet, self).__init__()
        self.conv1 = nn.Sequential(
            nn.Conv2d(1, 64, kernel_size=11, padding=5),
            nn.BatchNorm2d(64),
            nn.ReLU(),
        )
        self.conv2 = nn.Sequential(
            nn.Conv2d(64, 32, kernel_size=1, padding=0),
            nn.BatchNorm2d(32),
            nn.ReLU(),
        )
        self.conv3 = nn.Sequential(
            nn.Conv2d(32, 1, kernel_size=7, padding=3)
        )

        self.conv4 = nn.Sequential(
            nn.Conv2d(1, 64, kernel_size=11, padding=5),
            nn.BatchNorm2d(64),
            nn.ReLU(),
        )
        self.conv5 = nn.Sequential(
            nn.Conv2d(64, 32, kernel_size=1, padding=0),
            nn.BatchNorm2d(32),
            nn.ReLU(),
        )
        self.conv6 = nn.Sequential(
            nn.Conv2d(32, 1, kernel_size=7, padding=3)
        )
        self.conv7 = nn.Sequential(
            nn.Conv2d(1, 64, kernel_size=11, padding=5),
            nn.BatchNorm2d(64),
            nn.ReLU(),
        )
        self.conv8 = nn.Sequential(
            nn.Conv2d(64, 32, kernel_size=1, padding=0),
            nn.BatchNorm2d(32),
            nn.ReLU(),
        )
        self.conv9 = nn.Sequential(
            nn.Conv2d(32, 1, kernel_size=7, padding=3)
        )
        self.conv10 = nn.Sequential(
            nn.Conv2d(1, 64, kernel_size=11, padding=5),
            nn.BatchNorm2d(64),
            nn.ReLU(),
        )
        self.conv11 = nn.Sequential(
            nn.Conv2d(64, 32, kernel_size=1, padding=0),
            nn.BatchNorm2d(32),
            nn.ReLU(),
        )
        self.conv12 = nn.Sequential(
            nn.Conv2d(32, 1, kernel_size=7, padding=3)
        )
    def forward(self, x):
        identify1=x
        out = self.conv1(x)
        out = self.conv2(out)
        out = self.conv3(out) 
        out = out + identify1
        identify2=out
        out = self.conv4(out)
        out = self.conv5(out)
        out = self.conv6(out) 
        out = out + identify2
        identify3=out
        out = self.conv7(out)
        out = self.conv8(out)
        out = self.conv9(out) 
        out = out + identify3
        identify4=out
      
        out = self.conv10(out)
        out = self.conv11(out)
        out = self.conv12(out) 
        out = out + identify4

        return out
</code></pre>
<p>is it wrong?</p>
<p>It’s loss backward time takes hours.<br>
<em>Would you mind helping me with fixing my problems,please.</em></p>

albanD: <p>Hi,</p>
<p>You mean doing a single backward call take hours?<br>
Can you give a small code sample that reproduces that?</p>

lifeblack: <p>Thanks for your response.<br>
Yes. Doing a single backward call, takes approximately an hour.</p>
<pre><code class="lang-auto">**Instantiating the Network, Optimizer,and Loss Criterion:**
model=ResidualLearningNet()
model.apply(weights_init)
model.to(device)
optimizer=torch.optim.SGD(model.parameters(),lr=0.00001)
criterion=nn.MSELoss()

**Training**
start_time = datetime.now()
n_epochs=10
print_every=1
val_loss_min=np.Inf
for epoch in range(n_epochs):
  train_loss=0
  model.train()
  for iteration,(Xstage1,X_batch) in enumerate(Train_Loader):
    optimizer.zero_grad()
    output=model(Xstage1.to(device))
    loss=criterion(output,X_batch.to(device))
    retain_graph=True
    loss.backward(retain_graph=True)
    optimizer.step()
    train_loss+=loss.item()*X_batch.shape[0]
    if iteration % print_every==0:
      is_training=model.training
      val_loss=0
      model.eval()
      for (Xstage1_valid,X_valid_batch) in Valid_Loader:
        output=model(Xstage1_valid.to(device))
        loss=criterion(output,X_valid.to(device))
        val_loss+=loss.item()*data.shape[0]
      val_loss=val_loss/len(Valid_Loader.sampler)
      if val_loss&lt;=val_loss_min:
        print('\tValidation loss decreased ({:.6f}--&gt;{:.6f}). Saving Model...'.format(val_loss_min,val_loss))
        torch.save(model.state_dict(),'model_second_step.pt')
        val_loss_min=val_loss
      model.train(mode=is_training)   
  train_loss=train_loss/len(Train_Loader.sampler)
  print('Epoch: {} \tTraining loss:{:.6f}'.format(epoch+1,train_loss))
end_time = datetime.now()
print('Duration: {}'.format(end_time - start_time))
</code></pre>
<p>did you mean this part?</p>

albanD: <p>Thanks for the code.<br>
Here the timing is for the whole epoch. This is possible that it takes an hour depending on how big your dataset is (and how big your model is already).</p>

lifeblack: <p>thanks.<br>
A single backward takes an hour not one epoch.<br>
Is it OK?</p>

albanD: <p>I wrote the script below to time the backward and it says that it took ~120ms on my laptop.<br>
If you use your code above to time, you are timing the whole epoch, not just a single backward call.</p>
<pre><code class="lang-python">import torch
from torch import nn
import time

class ResidualLearningNet(nn.Module):
    def __init__(self):
        super(ResidualLearningNet, self).__init__()
        self.conv1 = nn.Sequential(
            nn.Conv2d(1, 64, kernel_size=11, padding=5),
            nn.BatchNorm2d(64),
            nn.ReLU(),
        )
        self.conv2 = nn.Sequential(
            nn.Conv2d(64, 32, kernel_size=1, padding=0),
            nn.BatchNorm2d(32),
            nn.ReLU(),
        )
        self.conv3 = nn.Sequential(
            nn.Conv2d(32, 1, kernel_size=7, padding=3)
        )

        self.conv4 = nn.Sequential(
            nn.Conv2d(1, 64, kernel_size=11, padding=5),
            nn.BatchNorm2d(64),
            nn.ReLU(),
        )
        self.conv5 = nn.Sequential(
            nn.Conv2d(64, 32, kernel_size=1, padding=0),
            nn.BatchNorm2d(32),
            nn.ReLU(),
        )
        self.conv6 = nn.Sequential(
            nn.Conv2d(32, 1, kernel_size=7, padding=3)
        )
        self.conv7 = nn.Sequential(
            nn.Conv2d(1, 64, kernel_size=11, padding=5),
            nn.BatchNorm2d(64),
            nn.ReLU(),
        )
        self.conv8 = nn.Sequential(
            nn.Conv2d(64, 32, kernel_size=1, padding=0),
            nn.BatchNorm2d(32),
            nn.ReLU(),
        )
        self.conv9 = nn.Sequential(
            nn.Conv2d(32, 1, kernel_size=7, padding=3)
        )
        self.conv10 = nn.Sequential(
            nn.Conv2d(1, 64, kernel_size=11, padding=5),
            nn.BatchNorm2d(64),
            nn.ReLU(),
        )
        self.conv11 = nn.Sequential(
            nn.Conv2d(64, 32, kernel_size=1, padding=0),
            nn.BatchNorm2d(32),
            nn.ReLU(),
        )
        self.conv12 = nn.Sequential(
            nn.Conv2d(32, 1, kernel_size=7, padding=3)
        )
    def forward(self, x):
        identify1=x
        out = self.conv1(x)
        out = self.conv2(out)
        out = self.conv3(out) 
        out = out + identify1
        identify2=out
        out = self.conv4(out)
        out = self.conv5(out)
        out = self.conv6(out) 
        out = out + identify2
        identify3=out
        out = self.conv7(out)
        out = self.conv8(out)
        out = self.conv9(out) 
        out = out + identify3
        identify4=out
      
        out = self.conv10(out)
        out = self.conv11(out)
        out = self.conv12(out) 
        out = out + identify4

        return out

model = ResidualLearningNet()

inp = torch.rand(2, 1, 64, 64)
out = model(inp)

start = time.time()
out.sum().backward()
print("Took: ", time.time() - start)
</code></pre>

lifeblack: <p>I’m very very thankful of your reply.<br>
But I cannot find out which part of the code takes so much time.<br>
one 128 batch of my data in your takes 1.5 seconds but in my code takes 1 hour.<br>
Is it because of the initialization of the network?</p>
<pre><code class="lang-auto">model=ResidualLearningNet()
model.apply(weights_init)
model.to(device)
optimizer=torch.optim.SGD(model.parameters(),lr=0.000001)
criterion=nn.MSELoss()
</code></pre>
<p>I wrote the code in google colab. If it helps, I can send you the link of my code.</p>

albanD: <p>How big is your dataset? If it one batch takes 1.5s and the whole thing takes 1h. It would mean 2400 batches. So for 10 epochs, that’s 240 batches per epoch. And so 30k samples in your dataset.<br>
Is that correct?</p>

lifeblack: <p>no,I said before that one BATCH takes one hour… <img src="https://discuss.pytorch.org/images/emoji/apple/pensive.png?v=9" title=":pensive:" class="emoji" alt=":pensive:"> <img src="https://discuss.pytorch.org/images/emoji/apple/pensive.png?v=9" title=":pensive:" class="emoji" alt=":pensive:"> <img src="https://discuss.pytorch.org/images/emoji/apple/pensive.png?v=9" title=":pensive:" class="emoji" alt=":pensive:"> <img src="https://discuss.pytorch.org/images/emoji/apple/frowning_face.png?v=9" title=":frowning_face:" class="emoji" alt=":frowning_face:"><br>
I have 663 batches so one epoch takes 663 hours <img src="https://discuss.pytorch.org/images/emoji/apple/disappointed.png?v=9" title=":disappointed:" class="emoji" alt=":disappointed:">  <img src="https://discuss.pytorch.org/images/emoji/apple/sob.png?v=9" title=":sob:" class="emoji" alt=":sob:"></p>

ptrblck: <p>As <a class="mention" href="/u/alband">@albanD</a> explained, your code profiles the complete training.<br>
Did you change the code to profile a single iteration?<br>
What does <a class="mention" href="/u/alband">@albanD</a>’s <a href="https://discuss.pytorch.org/t/pytorch-question-loss-backward-takes-too-much-time/80313/6">code</a> return, if you run it on your machine?</p>

lifeblack: <p>Thanks for your respond.<br>
<a class="mention" href="/u/alband">@albanD</a> 's code took 280 ms in my laptop. I cannot find out what out what is the difference between then and which part of my code is responsible to taking too much time of loss backward part.</p>
<p>Could you explain this part? I can not understand this part.</p>
<blockquote>
<p>Did you change the code to profile a single iteration?</p>
</blockquote>

ptrblck: <p>Your code profiles the complete training including all epochs:</p>
<pre><code class="lang-python">start_time = datetime.now()
for epoch in range(n_epochs):
  train_loss=0
  model.train()
  for iteration,(Xstage1,X_batch) in enumerate(Train_Loader):
    optimizer.zero_grad()
    ...
  print('Epoch: {} \tTraining loss:{:.6f}'.format(epoch+1,train_loss))
end_time = datetime.now()
</code></pre>
<p><code>start_time</code> is called before the first epoch and iteration is executed, while <code>end_time</code> is called after all epochs are done.</p>
<p><a class="mention" href="/u/alband">@albanD</a>’s code profiles just the backward call:</p>
<pre><code class="lang-python">start = time.time()
out.sum().backward()
print("Took: ", time.time() - start)
</code></pre>
<p>So not a single batch is taking one hour, but the complete training, i.e. 10 epochs.</p>

lifeblack: <p>I put a print code to know when a single loss backward finished. There was almost one hour gap between two output of the print code.<br>
I wrote the code in google colab. If it helps, I can send you the link of my code.</p>

ptrblck: <p>Sure, feel free to post the link here.</p>

lifeblack: <p>You can find my code here:<br>
<a href="https://colab.research.google.com/drive/1AfernpzyhNpxX6Y5gU31YwqVR_sisVZu?usp=sharing" rel="nofollow noopener">https://colab.research.google.com/drive/1AfernpzyhNpxX6Y5gU31YwqVR_sisVZu?usp=sharing</a></p>

ptrblck: <p>Thanks for the code.<br>
In the cells, which use <code>datetime</code> to profile the code, you are still starting the timer before the first epoch and stopping after all epochs were done.<br>
The output also shows this:</p>
<pre><code class="lang-python">Epoch: 100 	Training loss:0.020610
Duration: 0:07:08.332585
</code></pre>
<p>As you can see, 100 epochs were performed before the timer was stopped.</p>
<p>If you want to profile a single backward call, you would need to stick to the <a href="https://discuss.pytorch.org/t/pytorch-question-loss-backward-takes-too-much-time/80313/6">posted code</a>, i.e. start the timer before the backward pass, perform the backward pass, and stop the timer.<br>
If you are working with a GPU, you should additionally synchronize the code before starting and stopping the timer via <code>torch.cuda.synchronize()</code>.</p>

lifeblack: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a> … thanks for reply… I changed the timing part of the code and it showed that one iteration takes one hour…</p>
<pre><code class="lang-auto">start_time = datetime.now()
n_epochs=10
print_every=1
val_loss_min=np.Inf
for epoch in range(n_epochs):
  train_loss=0
  model.train()
  for iteration,(Xstage1,X_batch) in enumerate(Train_Loader):
    optimizer.zero_grad()
    output=model(Xstage1.to(device))
    loss=criterion(output,X_batch.to(device))
    retain_graph=True
    end_time = datetime.now()
    print('before:',end_time-start_time)
    loss.backward(retain_graph=True)
    end_time_back = datetime.now()
    print('backward',end_time_back-end_time)
    optimizer.step()
    train_loss+=loss.item()*X_batch.shape[0]
    if iteration % print_every==0:
      is_training=model.training
      val_loss=0
      model.eval()
      for (Xstage1_valid,X_valid_batch) in Valid_Loader:
        output=model(Xstage1_valid.to(device))
        loss=criterion(output,X_valid_batch.to(device))
        val_loss+=loss.item()*data.shape[0]
      val_loss=val_loss/len(Valid_Loader.sampler)
      if val_loss&lt;=val_loss_min:
        print('\tValidation loss decreased ({:.6f}--&gt;{:.6f}). Saving Model...'.format(val_loss_min,val_loss))
        torch.save(model.state_dict(),'/content/drive/My Drive/dataset/model_second_step.pt')
        val_loss_min=val_loss
      model.train(mode=is_training)   
  train_loss=train_loss/len(Train_Loader.sampler)
  print('Epoch: {} \tTraining loss:{:.6f}'.format(epoch+1,train_loss))
end_time = datetime.now()
print('Duration: {}'.format(end_time - start_time))
</code></pre>
<p>The output is:</p>
<pre><code class="lang-auto">before: 0:00:00.055158
backward 1:02:52.721540
</code></pre>
<p>one hour for a single iteration <img src="https://discuss.pytorch.org/images/emoji/apple/frowning_face.png?v=9" title=":frowning_face:" class="emoji" alt=":frowning_face:"></p>

ptrblck: <p>Could you post the complete output, please?<br>
You are still using other timers before the training starts and ends, so I’m not convinced that you are indeed profiling a single iteration.</p>

lifeblack: <p>I used google colab for running the code. After 5 iteration it stopped. I will run i again and post it again.</p>

lifeblack: <p>This is the code:</p>
<pre><code class="lang-auto">n_epochs=6
print_every=1
val_loss_min=np.Inf
for epoch in range(n_epochs):
  train_loss=0
  model.train()
  for iteration,(Xstage1,X_batch) in enumerate(Train_Loader):
    print('Iteration:',iteration,'started')
    torch.cuda.synchronize()
    strat_iteration=datetime.now()
    optimizer.zero_grad()
    output=model(Xstage1.to(device))
    loss=criterion(output,X_batch.to(device))
    retain_graph=True
    loss.backward(retain_graph=True)
    optimizer.step()
    train_loss+=loss.item()*X_batch.shape[0]
    torch.cuda.synchronize()
    end_iteration = datetime.now() 
    print('Iteration stopped.')
    print('Iteration time:',end_iteration-strat_iteration)
    if iteration % print_every==0:
      is_training=model.training
      val_loss=0
      model.eval()
      for (Xstage1_valid,X_valid_batch) in Valid_Loader:
        output=model(Xstage1_valid.to(device))
        loss=criterion(output,X_valid_batch.to(device))
        val_loss+=loss.item()*data.shape[0]
      val_loss=val_loss/len(Valid_Loader.sampler)
      if val_loss&lt;=val_loss_min:
        print('\tValidation loss decreased ({:.6f}--&gt;{:.6f}). Saving Model...'.format(val_loss_min,val_loss))
        torch.save(model.state_dict(),'/content/drive/My Drive/dataset/model_second_step.pt')
        val_loss_min=val_loss
      model.train(mode=is_training)   
  train_loss=train_loss/len(Train_Loader.sampler)
  print('Epoch: {} \tTraining loss:{:.6f}'.format(epoch+1,train_loss))
end_time = datetime.now()
print('Duration: {}'.format(end_time - start_time))

</code></pre>
<p>and the complete output is:</p>
<pre><code class="lang-auto">Iteration: 0 started
Iteration stopped.
Iteration time: 1:04:42.598970
	Validation loss decreased (inf--&gt;0.008777). Saving Model...
Iteration: 1 started
Iteration stopped.
Iteration time: 1:04:23.584298
	Validation loss decreased (0.008777--&gt;0.008767). Saving Model...
Iteration: 2 started
Iteration stopped.
Iteration time: 1:05:28.936429
	Validation loss decreased (0.008767--&gt;0.008762). Saving Model...
Iteration: 3 started
Iteration stopped.
Iteration time: 1:03:29.209401
	Validation loss decreased (0.008762--&gt;0.008758). Saving Model...
Iteration: 4 started
Iteration stopped.
Iteration time: 1:03:13.451551
	Validation loss decreased (0.008758--&gt;0.008754). Saving Model...
Iteration: 5 started
Iteration stopped.
Iteration time: 1:03:05.595077
	Validation loss decreased (0.008754--&gt;0.008751). Saving Model...
Iteration: 6 started
Iteration stopped.
Iteration time: 1:03:31.586236
	Validation loss decreased (0.008751--&gt;0.008749). Saving Model...
Iteration: 7 started
Iteration stopped.
Iteration time: 1:05:06.575140
Iteration: 8 started
</code></pre>