Compare layers' weights

username: sigeek
cooked: <p>Hi everyone!<br>
I’m a newbie to pytorch and I want to find a pretty way to compare layers’ weights between two models.<br>
I’m currently using an alexnet and I’m freezing all the layers exept for the convolutional layers.</p>
<pre><code class="lang-auto">net = alexnet(pretrained=True) 
#freeze all the layers
for param in net.parameters():
    param.requires_grad = False

# unfreeze only the fully connected layers
net.classifier[6] = nn.Linear(4096, NUM_CLASSES) 
for val in net.classifier:
  val.requires_grad = True
</code></pre>
<p>After the training phase, I save the best net (the one with the highest accuracy on the validation set) and creates some histograms with tensorboard.</p>
<pre><code class="lang-auto">basic_net = alexnet(pretrained=True)
tb = SummaryWriter(comment = "Compare Conv Layers")
j = 0
for i in [0,3,6,8,10]:
  tb.add_histogram("Pretrained conv layer", basic_net.features[i].weight.data.cpu().numpy(), j)
  j = j+1
tb.close

tb = SummaryWriter(comment = "Compare Conv Layers")
j = 0
for i in [0,3,6,8,10]:
  tb.add_histogram("My conv layer", best_net_3_2_1.features[i].weight.data.cpu().numpy(), j)
  j = j+1
tb.close

tb.flush()
</code></pre>
<p>With the index i I consider just the convolutional layers.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/b/2bd3f6546e4ca737a6eef10af905f3045a7c0baf.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/2bd3f6546e4ca737a6eef10af905f3045a7c0baf" title="Annotazione 2020-05-01 153248"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/b/2bd3f6546e4ca737a6eef10af905f3045a7c0baf_2_275x500.jpeg" alt="Annotazione 2020-05-01 153248" data-base62-sha1="6fIEO8n3iaqJC9wYt2pi0WRtV6L" width="275" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/b/2bd3f6546e4ca737a6eef10af905f3045a7c0baf_2_275x500.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/2/b/2bd3f6546e4ca737a6eef10af905f3045a7c0baf.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/2/b/2bd3f6546e4ca737a6eef10af905f3045a7c0baf.jpeg 2x" data-dominant-color="F5F4F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Annotazione 2020-05-01 153248</span><span class="informations">313×568 23.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
This is what I obtained in tensorboard, is there any better way to compare the weights?</p>

username: klory
cooked: <p>I don’t think displaying two histograms on the same figure is supported with tf-board, but you can have a third hist to show the difference between the two</p>

username: klory
cooked: <p>Another solution is try <a href="https://www.wandb.com/" rel="nofollow noopener">WandB</a> (free for personal use) instead of Tensorboard, I recently found it really handful</p>

username: sigeek
cooked: <p>Thanks for all the advices, I’m currently only displaying the difference but they are all zeros.</p>
<pre><code class="lang-auto">basic_net = alexnet(pretrained=True)
tb = SummaryWriter(comment = "Diff")
j = 0
for i in [0,3,6,8,10]:
  tb.add_histogram("Diff", basic_net.features[i].weight.data.cpu().numpy() - best_net_3_2_2.features[i].weight.data.cpu().numpy(), j)
  j = j+1
tb.close()

tb = SummaryWriter(comment = "Diffb")
j = 0
for i in [0,3,6,8,10]:
  tb.add_histogram("Diffb", basic_net.features[i].bias.data.cpu().numpy() - best_net_3_2_2.features[i].bias.data.cpu().numpy(), j)
  j = j+1
tb.close()
tb.flush()
</code></pre>
<p>Is that possible? Am I missing something?</p>