Why it takes the same time to calculate the gradient of the last layer and all the gradients？

cindybrain: <p>I want to save time by using the gradients of the last layer to replace that of all layers, however, I find it takes the same time  to calculate the gradient of the last layer and all the gradients.<br>
the code for calculating the gradient for the last layer:</p>
<pre><code class="lang-auto">    last_layer=list(net.children())[-1]
    for param in net.parameters():
        param.requires_grad = False 
    for param in last_layer.parameters():
        param.requires_grad = True 
    
    for i in range(len(inputs)):
        optimizer.zero_grad()
        loss[i].backward(retain_graph=True)
        last_grad_norm=0.0
        for name, para in last_layer.named_parameters():
            if 'weight' in name:
                last_grad_norm += para.grad.norm().cpu().item() 
        result.append(last_grad_norm)

</code></pre>
<p>the code for calculating the gradient for all layers:</p>
<pre><code class="lang-auto">    for i in range(len(inputs)):
        optimizer.zero_grad()
        loss[i].backward(retain_graph=True)
        all_grad_norm=0.0
        for name, para in last_layer.named_parameters():
            if 'weight' in name:
                all_grad_norm += para.grad.norm().cpu().item() 
        result.append(all_grad_norm)
</code></pre>
<p>I want to know why it cosumes the same time even that I only calculate gradients for partical layers. Meanwhile, I want to know how can I get the gradients for the last layer faster. I will be quite appreciate if someone can help me. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

ptrblck: <p>If the input doesn’t require gradients, calculating the gradients for a single layer should be faster than all layers.<br>
You code doesn’t show any profiling so I cannot comment on the correctness of it.<br>
Note that, in case you are using a GPU, CUDA operations are asynchronous, so you would have to synchronize the code before starting and stopping the timer or use the <code>torch.utils.benchmark</code> utility from the nightly binaries.<br>
That being said, you are currently also synchronizing your “production” code by pushing the gradient norm to the CPU and calling <code>item()</code> on it.</p>

cindybrain: <p>Thanks for your reply!<br>
I modify the code and use the time.time() function to record the time<br>
the code is as follows:</p>
<pre><code class="lang-auto">    last_layer=list(net.children())[-1]
    for param in net.parameters():
        param.requires_grad = False 
    for param in last_layer.parameters():
        param.requires_grad = True

    torch.cuda.synchronize()
    start_time=time.time()
    for i in range(len(inputs)):
        optimizer.zero_grad()
        loss[i].backward(retain_graph=True)
        last_grad_norm=0.0
        for name, para in last_layer.named_parameters():
            if 'weight' in name:
                last_grad_norm += para.grad.norm().cpu().item() 
        # print("last grad norm", last_grad_norm) 
        result.append(last_grad_norm)
    torch.cuda.synchronize()
    print("the time for computing the gradient for the last layer:",time.time()-start_time)
</code></pre>
<pre><code class="lang-auto">    for param in net.parameters():
        param.requires_grad = True
    torch.cuda.synchronize()
    start_time=time.time()
    for i in range(len(inputs)):
        optimizer.zero_grad()
        loss[i].backward(retain_graph=True)
        all_grad_norm=0.0
        for name, para in last_layer.named_parameters():
            if 'weight' in name:
                all_grad_norm += para.grad.norm().cpu().item() 
        result.append(all_grad_norm)
    torch.cuda.synchronize()   
    print("the time for computing the gradient for all layers:",time.time()-start_time)
</code></pre>
<p>the result is as follows:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/b/dbe2bbfb0c27884c7fab538739008fdffa3eeff8.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/dbe2bbfb0c27884c7fab538739008fdffa3eeff8" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/b/dbe2bbfb0c27884c7fab538739008fdffa3eeff8_2_690x136.jpeg" alt="image" data-base62-sha1="vnclcK0TBmpLebJkChI9154WUDm" width="690" height="136" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/b/dbe2bbfb0c27884c7fab538739008fdffa3eeff8_2_690x136.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/d/b/dbe2bbfb0c27884c7fab538739008fdffa3eeff8_2_1035x204.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/d/b/dbe2bbfb0c27884c7fab538739008fdffa3eeff8_2_1380x272.jpeg 2x" data-dominant-color="3B3B3B"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1618×320 153 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>The cost time is almost similar.</p>
<p>Besides, because I am a new learner of Pytorch, I don’t know how to compute the gradient of loss about the input of the last layer, which include the softmax,bn and the last layer- conv. Could you give me an example or reference. Thanks so much! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>