Tracing a graph of layers in a model

username: Vedant_Dalimkar
cooked: <p>Trying to implement structured filter pruning from scratch. Is there anyway to get the dependancy graph of layers in a model?</p>
<p>I will need a graph because if i remove a filter from a nn.conv2d layer, I will need to modify the structure of all downstream layers which are directly dependant on the layer which I am pruning.</p>
<p>Any straightforward way to do this in pytorch?</p>

username: anantguptadbl
cooked: <p><a class="mention" href="/u/vedant_dalimkar">@Vedant_Dalimkar</a> The following post can be helpful</p>
<aside class="quote quote-modified" data-post="1" data-topic="40542">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/p/f1d935/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/how-to-drop-specific-filters-in-some-cnn-layers/40542">How to drop specific filters in some CNN layers</a> <a class="badge-category__wrapper " href="/c/vision/5"><span data-category-id="5" style="--category-badge-color: #AB9364; --category-badge-text-color: #FFFFFF;" data-drop-close="true" class="badge-category " title="Topics related to either pytorch/vision or vision research related topics"><span class="badge-category__name">vision</span></span></a>
  </div>
  <blockquote>
    I have read the 2019 CVPR paper:RePr: Improved Training of Convolutional Filters and want to implement its method. First, we need to train a CNN several iterations. Then we can use a metric algorithm to rank all filters and drop the 30% least important filters to continue to train a few iterations just like pruning. After that, re-initialize the filters  that are dropped before and continue to train the networks. We modify the training process by cyclically removing redundant ﬁlters, retraining …
  </blockquote>
</aside>


username: Vedant_Dalimkar
cooked: <p>This didn’t help me.</p>
<p>Let’s say I want to remove a filter from a nn.conv2d layer, it has ‘out_channels’ number of filters each having ‘in_channels’ channel dimension. Now if I remove a filter from this conv2d layer, I will need to change the ‘in_channels’ argument for all the conv2d layers (might be more than 1 downstream dependant  layer due to residual connections) that are dependant on this ‘changed’ conv2d layer.</p>
<p>Hope that makes my question clearer.</p>

username: anantguptadbl
cooked: <p><a class="mention" href="/u/vedant_dalimkar">@Vedant_Dalimkar</a>  Understood<br>
What you want, that needs to be coded up</p>
<ol>
<li>Create a new conv2d layer and copy the parameters (weights and biases) from the old layer, while excluding the filters that you want to prune</li>
<li>Create new layers before and after this layer that ensures that upstream and downstream layers are not impacted</li>
<li>Replace the old conv2d layer with this new Sequential</li>
</ol>
<p>All existing pruning libraries, will mask the weights but not completely remove the filter<br>
e.g.<br>
<a href="https://pytorch.org/docs/stable/generated/torch.nn.utils.prune.ln_structured.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/generated/torch.nn.utils.prune.ln_structured.html</a></p>

username: Vedant_Dalimkar
cooked: <p>At runtime we don’t know which filter in which layer will be pruned so that is why having a dependency graph is a must here.</p>
<p>For example take the last conv layer in the contracting portion of Unet. Its output will be passed to the next layer as well as to a layer in the expanding portion of Unet. Your approach won’t work here as the pruned layer and all possible downstream layer are not in a single nn.Sequential.</p>

username: anantguptadbl
cooked: <p><a class="mention" href="/u/vedant_dalimkar">@Vedant_Dalimkar</a> okay. Maybe i am not able to understand why do you want to touch other layers of a model apart from the layer that you are pruning</p>
<p>With regards to seeing the dependency graph, you can refer to this post</p><aside class="onebox stackexchange" data-onebox-src="https://stackoverflow.com/questions/52468956/how-do-i-visualize-a-net-in-pytorch">
  <header class="source">

      <a href="https://stackoverflow.com/questions/52468956/how-do-i-visualize-a-net-in-pytorch" target="_blank" rel="noopener nofollow ugc">stackoverflow.com</a>
  </header>

  <article class="onebox-body">
      <a href="https://stackoverflow.com/users/1436508/raaj" target="_blank" rel="noopener nofollow ugc">
    <img alt="raaj" src="https://lh3.googleusercontent.com/-p4GumfwGZus/AAAAAAAAAAI/AAAAAAAAAAA/AGDgw-i9vW1oPm_00XtAVO06i_X9sWp2sA/mo/photo.jpg?sz=256" class="thumbnail onebox-avatar" width="" height="">
  </a>

<h4>
  <a href="https://stackoverflow.com/questions/52468956/how-do-i-visualize-a-net-in-pytorch" target="_blank" rel="noopener nofollow ugc">How do I visualize a net in Pytorch?</a>
</h4>

<div class="tags">
  <strong>python, pytorch</strong>
</div>

<div class="date">
  asked by
  
  <a href="https://stackoverflow.com/users/1436508/raaj" target="_blank" rel="noopener nofollow ugc">
    raaj
  </a>
  on <a href="https://stackoverflow.com/questions/52468956/how-do-i-visualize-a-net-in-pytorch" target="_blank" rel="noopener nofollow ugc">06:15PM - 23 Sep 18 UTC</a>
</div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>It has multiple examples with multiple libraries</p>
<p>Happy coding</p>

username: Vedant_Dalimkar
cooked: <aside class="quote no-group" data-username="Vedant_Dalimkar" data-post="3" data-topic="205864">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/vedant_dalimkar/48/67190_2.png" class="avatar"> Vedant_Dalimkar:</div>
<blockquote>
<p>Let’s say I want to remove a filter from a nn.conv2d layer, it has ‘out_channels’ number of filters each having ‘in_channels’ channel dimension. Now if I remove a filter from this conv2d layer, I will need to change the ‘in_channels’ argument for all the conv2d layers (might be more than 1 downstream dependant layer due to residual connections) that are dependant on this ‘changed’ conv2d layer</p>
</blockquote>
</aside>
<p>Read this again, let me know if it’s still unclear.</p>