ResNet50 - Computing Sparsity

username: grid_world
cooked: <p>I have trained a ResNet-50 model on CIFAR-10 using transfer learning with some modifications. After which, I am using torch.nn.utils.prune to prune weights using global magnitude pruning in an iterative manner. The model architecture is as follows:</p>
<pre><code class="lang-auto">print(fine_tuned_model)
ResNet(
  (conv1): Conv2d(3, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
  (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (relu): ReLU(inplace=True)
  (maxpool): MaxPool2d(kernel_size=3, stride=2, padding=1, dilation=1, ceil_mode=False)
  (layer1): Sequential(
    (0): Bottleneck(
      (conv1): Conv2d(64, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)
      (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (conv3): Conv2d(64, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
      (bn3): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu): ReLU(inplace=True)
      (downsample): Sequential(
        (0): Conv2d(64, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
    )
    (1): Bottleneck(
      (conv1): Conv2d(256, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)
      (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (conv3): Conv2d(64, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
      (bn3): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu): ReLU(inplace=True)
    )
    (2): Bottleneck(
      (conv1): Conv2d(256, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)
      (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (conv3): Conv2d(64, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
      (bn3): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu): ReLU(inplace=True)
    )
  )
  (layer2): Sequential(
    (0): Bottleneck(
      (conv1): Conv2d(256, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
      (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (conv3): Conv2d(128, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
      (bn3): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu): ReLU(inplace=True)
      (downsample): Sequential(
        (0): Conv2d(256, 512, kernel_size=(1, 1), stride=(2, 2), bias=False)
        (1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
    )
    (1): Bottleneck(
      (conv1): Conv2d(512, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
      (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (conv3): Conv2d(128, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
      (bn3): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu): ReLU(inplace=True)
    )
    (2): Bottleneck(
      (conv1): Conv2d(512, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
      (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (conv3): Conv2d(128, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
      (bn3): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu): ReLU(inplace=True)
    )
    (3): Bottleneck(
      (conv1): Conv2d(512, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)
      (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (conv3): Conv2d(128, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
      (bn3): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu): ReLU(inplace=True)
    )
  )
  (layer3): Sequential(
    (0): Bottleneck(
      (conv1): Conv2d(512, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
      (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (conv3): Conv2d(256, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False)
      (bn3): BatchNorm2d(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu): ReLU(inplace=True)
      (downsample): Sequential(
        (0): Conv2d(512, 1024, kernel_size=(1, 1), stride=(2, 2), bias=False)
        (1): BatchNorm2d(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
    )
    (1): Bottleneck(
      (conv1): Conv2d(1024, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
      (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (conv3): Conv2d(256, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False)
      (bn3): BatchNorm2d(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu): ReLU(inplace=True)
    )
    (2): Bottleneck(
      (conv1): Conv2d(1024, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
      (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (conv3): Conv2d(256, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False)
      (bn3): BatchNorm2d(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu): ReLU(inplace=True)
    )
    (3): Bottleneck(
      (conv1): Conv2d(1024, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
      (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (conv3): Conv2d(256, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False)
      (bn3): BatchNorm2d(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu): ReLU(inplace=True)
    )
    (4): Bottleneck(
      (conv1): Conv2d(1024, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
      (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (conv3): Conv2d(256, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False)
      (bn3): BatchNorm2d(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu): ReLU(inplace=True)
    )
    (5): Bottleneck(
      (conv1): Conv2d(1024, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)
      (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (conv3): Conv2d(256, 1024, kernel_size=(1, 1), stride=(1, 1), bias=False)
      (bn3): BatchNorm2d(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu): ReLU(inplace=True)
    )
  )
  (layer4): Sequential(
    (0): Bottleneck(
      (conv1): Conv2d(1024, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
      (bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (conv3): Conv2d(512, 2048, kernel_size=(1, 1), stride=(1, 1), bias=False)
      (bn3): BatchNorm2d(2048, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu): ReLU(inplace=True)
      (downsample): Sequential(
        (0): Conv2d(1024, 2048, kernel_size=(1, 1), stride=(2, 2), bias=False)
        (1): BatchNorm2d(2048, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
    )
    (1): Bottleneck(
      (conv1): Conv2d(2048, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
      (bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (conv3): Conv2d(512, 2048, kernel_size=(1, 1), stride=(1, 1), bias=False)
      (bn3): BatchNorm2d(2048, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu): ReLU(inplace=True)
    )
    (2): Bottleneck(
      (conv1): Conv2d(2048, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)
      (bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (conv3): Conv2d(512, 2048, kernel_size=(1, 1), stride=(1, 1), bias=False)
      (bn3): BatchNorm2d(2048, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu): ReLU(inplace=True)
    )
  )
  (avgpool): AdaptiveAvgPool2d(output_size=(1, 1))
  (fc): Linear(in_features=2048, out_features=10, bias=True)
)
</code></pre>
<p>To count the number of pruned connections (or, number of zeroes), number of total connections and the resulting sparsity, the following two functions are defined:</p>
<pre><code class="lang-auto">def measure_module_sparsity(module, weight = True, bias = False, use_mask = False):

    num_zeros = 0
    num_elements = 0

    if use_mask == True:
        for buffer_name, buffer in module.named_buffers():
            if "weight_mask" in buffer_name and weight == True:
                num_zeros += torch.sum(buffer == 0).item()
                num_elements += buffer.nelement()
            if "bias_mask" in buffer_name and bias == True:
                num_zeros += torch.sum(buffer == 0).item()
                num_elements += buffer.nelement()
    else:
        for param_name, param in module.named_parameters():
            if "weight" in param_name and weight == True:
                num_zeros += torch.sum(param == 0).item()
                num_elements += param.nelement()
            if "bias" in param_name and bias == True:
                num_zeros += torch.sum(param == 0).item()
                num_elements += param.nelement()

    sparsity = num_zeros / num_elements

    return num_zeros, num_elements, sparsity



def measure_global_sparsity(
    model, weight = True,
    bias = False, conv2d_use_mask = False,
    linear_use_mask = False):

    num_zeros = 0
    num_elements = 0

    for module_name, module in model.named_modules():

        if isinstance(module, torch.nn.Conv2d):
            module_num_zeros, module_num_elements, _ = measure_module_sparsity(
                module, weight=weight, bias=bias, use_mask=conv2d_use_mask)
            num_zeros += module_num_zeros
            num_elements += module_num_elements

        elif isinstance(module, torch.nn.Linear):
            module_num_zeros, module_num_elements, _ = measure_module_sparsity(
                module, weight=weight, bias=bias, use_mask=linear_use_mask)
            num_zeros += module_num_zeros
            num_elements += module_num_elements

    sparsity = num_zeros / num_elements

    return num_zeros, num_elements, sparsity
</code></pre>
<p>To call the function-</p>
<pre><code class="lang-auto">num_zeros, num_elements, sparsity = measure_global_sparsity(
    fine_tuned_model, weight = True,
    bias = False, conv2d_use_mask = True,
    linear_use_mask = False)
</code></pre>
<p>this gives the error:</p>
<blockquote>
<p>ZeroDivisionError                         Traceback (most recent call last)<br>
 in <br>
----&gt; 1 num_zeros, num_elements, sparsity = measure_global_sparsity(<br>
2     fine_tuned_model, weight = True,<br>
3     bias = False, conv2d_use_mask = True,<br>
4     linear_use_mask = False)<br>
5</p>
<p> in measure_global_sparsity(model, weight, bias, conv2d_use_mask, linear_use_mask)<br>
10<br>
11         if isinstance(module, torch.nn.Conv2d):<br>
—&gt; 12             module_num_zeros, module_num_elements, _ = measure_module_sparsity(<br>
13                 module, weight=weight, bias=bias, use_mask=conv2d_use_mask)<br>
14             num_zeros += module_num_zeros</p>
<p> in measure_module_sparsity(module, weight, bias, use_mask)<br>
21                 num_elements += param.nelement()<br>
22<br>
—&gt; 23     sparsity = num_zeros / num_elements<br>
24<br>
25     return num_zeros, num_elements, sparsity</p>
<p>ZeroDivisionError: division by zero</p>
</blockquote>
<p>What’s the problem OR is there another way to count the: number of zeroes and the total number of parameters within this ResNet-50 model?</p>

username: randinoo
cooked: <p>why you put 64 of conv1??instead of 10 or 20 etc</p>

username: KFrank
cooked: <p>Hi Arjun!</p>
<aside class="quote no-group quote-modified" data-username="grid_world" data-post="1" data-topic="120956" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/grid_world/48/33204_2.png" class="avatar"> grid_world:</div>
<blockquote>
<pre><code class="lang-auto">def measure_module_sparsity(module, weight = True, bias = False, use_mask = False):

    num_zeros = 0
    num_elements = 0

    if use_mask == True:
        for buffer_name, buffer in module.named_buffers():
            ...
    else:
        for param_name, param in module.named_parameters():
            ...

    sparsity = num_zeros / num_elements

    return num_zeros, num_elements, sparsity
</code></pre>
<blockquote>
<pre><code> 22
 —&gt; 23 sparsity = num_zeros / num_elements
 24
 25     return num_zeros, num_elements, sparsity
</code></pre>
<p>ZeroDivisionError: division by zero</p>
</blockquote>
<p>What’s the problem … ?</p>
</blockquote>
</aside>
<p>If you call <code>measure_module_sparsity()</code> on a <code>Module</code> such as <code>ReLU</code><br>
that contains neither buffers nor parameters, then <code>num_elements</code> (and<br>
<code>num_zeros</code>) will remain zero (because they never get incremented)<br>
so you get the <code>ZeroDivisionError</code>.</p>
<p>Best.</p>
<p>K. Frank</p>

username: grid_world
cooked: <p>I am referring to <a href="https://arxiv.org/abs/1512.03385" rel="noopener nofollow ugc">ResNet paper</a> by Kaiming He et al. where in Table 1, the first conv layer has 64 filters in it. I just changed filter size from 7 to 3 and stride from 2 to 1.</p>
<p>Probably, removing a max pool layer might also make sense since CIFAR-10 dataset has small images to begin with which don’t need spatial reduction right after the first conv layer.</p>

username: grid_world
cooked: <p>I don’t think that “measure_module_sparsity()” function will get called on a module such as “ReLU” because within “measure_global_sparsity()” function, the for loop:</p>
<pre><code class="lang-auto">for module_name, module in model.named_modules():
    if isinstance(module, torch.nn.Conv2d):
        module_num_zeros, module_num_elements, _ = measure_module_sparsity(
                module, weight=weight, bias=bias, use_mask=conv2d_use_mask)
        num_zeros += module_num_zeros
        num_elements += module_num_elements
</code></pre>
<p>will make sure that “measure_module_sparsity()” is only called for Conv2d and/or Linear layers.</p>
<p>At least, that’s what my understanding of the code is.</p>

username: KFrank
cooked: <p>Hi Arjun!</p>
<aside class="quote no-group" data-username="grid_world" data-post="5" data-topic="120956" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/grid_world/48/33204_2.png" class="avatar"> grid_world:</div>
<blockquote>
<p>I don’t think that “measure_module_sparsity()” function will get called on a module such as “ReLU”</p>
</blockquote>
</aside>
<p>Yes, you are correct.  My mistake.</p>
<p>All I can say is that your code works for me (on a stock torchvision<br>
<code>resnet50</code>).</p>
<p>It is possible to instantiate a degenerate <code>Linear</code> (or <code>Conv2d</code>) that has<br>
no elements in its parameters, but that would be weird, and doesn’t<br>
happen in torchvision’s <code>resnet50</code> and does not appear to be happening<br>
in your model.</p>
<p>All I can suggest is that you check (and tell us) what version of pytorch<br>
you are using, and instrument your code so that it prints out the name<br>
(and details) of any module for which <code>num_elements</code> is zero before<br>
performing the division.</p>
<p>Here is a script that runs your function on a stock <code>resnet50</code> and also<br>
illustrates a degenerate <code>Linear</code>:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
print (torch.__version__)

import torchvision
print (torchvision.__version__)

_ = torch.manual_seed (2021)

def measure_module_sparsity(module, weight = True, bias = False, use_mask = False):
    
    num_zeros = 0
    num_elements = 0
    
    if use_mask == True:
        for buffer_name, buffer in module.named_buffers():
            if "weight_mask" in buffer_name and weight == True:
                num_zeros += torch.sum(buffer == 0).item()
                num_elements += buffer.nelement()
            if "bias_mask" in buffer_name and bias == True:
                num_zeros += torch.sum(buffer == 0).item()
                num_elements += buffer.nelement()
    else:
        for param_name, param in module.named_parameters():
            if "weight" in param_name and weight == True:
                num_zeros += torch.sum(param == 0).item()
                num_elements += param.nelement()
            if "bias" in param_name and bias == True:
                num_zeros += torch.sum(param == 0).item()
                num_elements += param.nelement()
    
    sparsity = num_zeros / num_elements
    
    return num_zeros, num_elements, sparsity


def measure_global_sparsity(
    model, weight = True,
    bias = False, conv2d_use_mask = False,
    linear_use_mask = False):
    
    num_zeros = 0
    num_elements = 0
    
    for module_name, module in model.named_modules():
        
        if isinstance(module, torch.nn.Conv2d):
            module_num_zeros, module_num_elements, _ = measure_module_sparsity(
                module, weight=weight, bias=bias, use_mask=conv2d_use_mask)
            num_zeros += module_num_zeros
            num_elements += module_num_elements
        
        elif isinstance(module, torch.nn.Linear):
            module_num_zeros, module_num_elements, _ = measure_module_sparsity(
                module, weight=weight, bias=bias, use_mask=linear_use_mask)
            num_zeros += module_num_zeros
            num_elements += module_num_elements
    
    sparsity = num_zeros / num_elements
    
    return num_zeros, num_elements, sparsity

mod = torchvision.models.resnet50()

print ('type (mod) =', type (mod))
print ('measure_global_sparsity (mod) =', measure_global_sparsity (mod))

lin = torch.nn.Linear (1, 0)
print ('lin =', lin)
print ('calling measure_global_sparsity (lin) ...')
measure_global_sparsity (lin)
</code></pre>
<p>And here is its output:</p>
<pre><code class="lang-plaintext">1.9.0.dev20210504
0.10.0.dev20210504
type (mod) = &lt;class 'torchvision.models.resnet.ResNet'&gt;
measure_global_sparsity (mod) = (4, 25502912, 1.5684483403307042e-07)
lin = Linear(in_features=1, out_features=0, bias=True)
calling measure_global_sparsity (lin) ...
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  File "&lt;string&gt;", line 70, in &lt;module&gt;
  File "&lt;string&gt;", line 53, in measure_global_sparsity
  File "&lt;string&gt;", line 31, in measure_module_sparsity
ZeroDivisionError: division by zero
</code></pre>
<p>Best.</p>
<p>K. Frank</p>

username: grid_world
cooked: <p>PyTorch Version: 1.8.1<br>
Torchvision Version: 0.9.1</p>
<p>Good idea. Let me try your suggestion and get back.</p>
<p>I am using Global, unstructured, absolute magnitude &amp; iterative pruning by using “torch.nn.utils.prune” for which computing global sparsity is important. The algo used involves using a trained model as follows:</p>
<ol>
<li>
<p>prune p% of smallest magnitude weights globally (p = 20%)</p>
</li>
<li>
<p>Fine-tune for n epochs to recover from the damage caused due to pruning</p>
</li>
</ol>