Is there a way to visualize the gradient path of the back propagation of the entire network

I-Love-U: <p>Hello.</p>
<p>Now, my network has two branches, one of which is the normal ResNet50 and the other branch is forked from the third convolution block of ResNet50. In the latter branch, I set some operations, one of which is <strong>as follows</strong>.</p>
<pre><code class="lang-python"># https://github.com/yanx27/3DGNN_pytorch/blob/b5e0188e56f926cff9b2c9a68bedf42cb3b42d2f/models.py#L340

    # adapted from https://discuss.pytorch.org/t/build-your-own-loss-function-in-pytorch/235/6
    # (x - y)^2 = x^2 - 2*x*y + y^2
    def get_knn_indices(self, batch_mat, k):
        r = torch.bmm(batch_mat, batch_mat.permute(0, 2, 1)) 
        N = r.size()[0]
        HW = r.size()[1]
        if self.use_gpu:
            batch_indices = torch.zeros((N, HW, k)).cuda()
        else:
            batch_indices = torch.zeros((N, HW, k))
        for idx, val in enumerate(r):
            # get the diagonal elements
            diag = val.diag().unsqueeze(0)
            diag = diag.expand_as(val)
            # compute the distance matrix
            D = (diag + diag.t() - 2 * val).sqrt()
            topk, indices = torch.topk(D, k=k, largest=False)
            batch_indices[idx] = indices.data
        return batch_indices
</code></pre>
<p>I think <code>__get_knn_indices</code> is non-differentiable, which may cause the parameters of this branch not to be updated when backpropagating.</p>
<p>So, I have some questions:</p>
<ol>
<li>How can I tell if my thoughts are correct?</li>
<li>Is there a way to visualize <strong>the gradient path of the back propagation of the entire network</strong>? If there is any, it is estimated that my first problem can be easily solved.</li>
</ol>

JuanFMontesinos: <p>Meh, there are some works which allows you to plot the graph but you can get masive huge graphs.</p>
<p>Why don’t you just visualize gradients? if there is a detach you will see None instead of a number</p>

I-Love-U: <p>Thanks for your reply.</p>
<p>According to your advice, I find <code>tensorboardX</code> can do it:</p>
<aside class="onebox githubissue">
  <header class="source">
      <a href="https://github.com/lanpa/tensorboardX/issues/212" target="_blank" rel="nofollow noopener">github.com/lanpa/tensorboardX</a>
  </header>
  <article class="onebox-body">
    <a href="https://github.com/HyoungWooPark" rel="nofollow noopener">
<img src="https://avatars1.githubusercontent.com/u/12387332?v=2&amp;s=96" class="thumbnail onebox-avatar" width="60" height="60">
</a>

<h4><a href="https://github.com/lanpa/tensorboardX/issues/212" target="_blank" rel="nofollow noopener">Issue: How to visualize gradient with tensorboardX in pytorch</a></h4>

<div class="date" style="margin-top:10px;">
	<div class="user" style="margin-top:10px;">
	opened by <a href="https://github.com/HyoungWooPark" target="_blank" rel="nofollow noopener">HyoungWooPark</a>
	on <a href="https://github.com/lanpa/tensorboardX/issues/212" target="_blank" rel="nofollow noopener">2018-08-15</a>
	</div>
	<div class="user">
	closed by <a href="https://github.com/lanpa" target="_blank" rel="nofollow noopener">lanpa</a>
	on <a href="https://github.com/lanpa/tensorboardX/issues/212" target="_blank" rel="nofollow noopener">2018-08-20</a>
	</div>
</div>

<pre class="content" style="white-space: pre-wrap;">I wonder how to visualize gradient by histogram with tensorboardX in pytorch</pre>

<div class="labels">
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>I’ll try it. Thank you again.</p>

drb12: <p>I am working on implementing this as well. At what point during the training should you check for the gradient?</p>
<p>Currently, I am checking at the end of each epoch by iterating through my models parameters and calling the variable <code>.grad</code> As shown in code below. However, for some reason when I visualize it in Tensorboard all my layers have zero gradients, even though the histograms show that the weights and bias are changing.</p>
<pre><code class="lang-auto">loss.backward()
optimizer.step()
optimizer.zero_grad()
for tag, parm in model.named_parameters:
     writer.add_histogram(tag, parm.grad.data.cpu().numpy(), epoch)
</code></pre>

I-Love-U: <p>I think you’re saying that this method is not accurate for observing gradients.<br>
I hope I get it right.</p>
<p>I haven’t tried yet, but the question you’re talking about is really interesting, so here’s the question. How should we observe the gradient?</p>

drevicko: <p>Perhaps you should be calling <code>add_histogram</code> before <code>.zero_grad()</code> … Calling it after, I’d expect to get zeros!</p>

dmitrysarov: <p><a class="mention" href="/u/drb12">@drb12</a> you have a typo in your code.</p>
<pre><code class="lang-auto">model.named_parameters
</code></pre>
<p>is a method, so to iterate over parameters you should call it</p>
<pre><code class="lang-auto">model.named_parameters()
</code></pre>

sharjeel: <p>you can use <code>BBDecoder</code> to visualize model gradients along with other metrics necessary for analyzing the model.</p>
<pre><code class="lang-auto">from BBDecoder import Master_analyzer

model = resnet50().to(device)
wrapped_model = Master_analyzer(model, save_path, input_size)

for data in tqdm(testloader):
    inputs, labels = data
    inputs, labels = inputs.to(device), labels.to(device)
    
    outputs = wrapped_model.forward_propagation(inputs)
    loss = F.cross_entropy(outputs, labels)

    optimizer.zero_grad()
    wrapped_model.backward_propagation(loss, collect_grads = True, layer_inds = [0, 1, 2, 3, 4, 5, 6])
    optimizer.step()
    losses.append(loss.item())
    
print('Epoch: ', epoch)
print('Average Loss: ', sum(losses)/len(losses))
wrapped_model.save_collected_grads(ep = epoch, save_folder)

</code></pre>