Pruning with "torch.nn.utils.prune" module - no pruning seen

grid_world: <p>I have been trying to implement pruning using " `torch.nn.utils.prune" according to the tutorial <a href="https://pytorch.org/tutorials/intermediate/pruning_tutorial.html#pruning-multiple-parameters-in-a-model" rel="noopener nofollow ugc">here</a>. For experimenting purposes, I am using LeNet-300-100 dense neural network with MNIST dataset which can be accessed <a href="https://github.com/arjun-majumdar/Neural_Network_Pruning/blob/main/Iterative_Pruning_LeNet300_PyTorch.ipynb" rel="noopener nofollow ugc">here</a>.</p>
<pre><code class="lang-auto"># Pruning multiple parameters-
# Prune multiple parameters/layers in a given model-
for name, module in best_model.named_modules():
    # prune 20% of weights/connections in for all hidden layaers-
    if isinstance(module, torch.nn.Linear) and name != 'output':
        prune.l1_unstructured(module = module, name = 'weight', amount = 0.2)
    
    # prune 10% of weights/connections for output layer-
    elif isinstance(module, torch.nn.Linear) and name == 'output':
        prune.l1_unstructured(module = module, name = 'weight', amount = 0.1)


# Sanity check: verify that all of the defined pruning exists as masks-
print(dict(best_model.named_buffers()).keys())
# dict_keys(['fc1.weight_mask', 'fc2.weight_mask', 'output.weight_mask'])
</code></pre>
<p>I then train the model with the following code snippet:</p>
<pre><code class="lang-auto"># Training loop-
for epoch in range(num_epochs):
    running_loss = 0.0
    running_corrects = 0.0
    
    if loc_patience &gt;= patience:
        print("\n'EarlyStopping' called!\n")
        break

    running_loss, running_corrects = train_model(best_model, train_loader)
  
    epoch_loss = running_loss / len(train_dataset)
    epoch_acc = running_corrects.double() / len(train_dataset)
    # epoch_acc = 100 * running_corrects / len(trainset)
    # print(f"\nepoch: {epoch + 1} training loss = {epoch_loss:.4f}, training accuracy = {epoch_acc * 100:.2f}%\n")

    running_loss_val, correct, total = test_model(best_model, test_loader)

    epoch_val_loss = running_loss_val / len(test_dataset)
    val_acc = 100 * (correct / total)
    # print(f"\nepoch: {epoch + 1} training loss = {epoch_loss:.4f}, training accuracy = {epoch_acc * 100:.2f}%, val_loss = {epoch_val_loss:.4f} &amp; val_accuracy = {val_acc:.2f}%\n")

    print(f"\nepoch: {epoch + 1} training loss = {epoch_loss:.4f}, training accuracy = {epoch_acc * 100:.2f}%, val_loss = {epoch_val_loss:.4f} &amp; val_accuracy = {val_acc:.2f}%")

    curr_params = count_params(best_model)
    print(f"Number of parameters = {curr_params}\n")
    
    percentage_pruned = ((orig_params - curr_params.numpy()) / orig_params * 100).numpy()
    
    # Code for manual Early Stopping:
    # if np.abs(epoch_val_loss &lt; best_val_loss) &gt;= minimum_delta:
    if (epoch_val_loss &lt; best_val_loss) and np.abs(epoch_val_loss - best_val_loss) &gt;= minimum_delta:
        # print(f"epoch_val_loss = {epoch_val_loss:.4f}, best_val_loss = {best_val_loss:.4f}")
        
        # update 'best_val_loss' variable to lowest loss encountered so far-
        best_val_loss = epoch_val_loss
        
        # reset 'loc_patience' variable-
        loc_patience = 0
        
        print(f"\nSaving model with lowest val_loss = {epoch_val_loss:.4f}")
        
        # Save trained model with validation accuracy-
        # torch.save(model.state_dict, f"LeNet-300-100_Trained_{val_acc}.pth")
        torch.save(best_model.state_dict(), f"LeNet-300-100_{percentage_pruned:.2f}.pth")
        
    else:  # there is no improvement in monitored metric 'val_loss'
        loc_patience += 1  # number of epochs without any improvement
</code></pre>
<p>The output is:</p>
<blockquote>
<p>epoch: 1 training loss = 0.0266, training accuracy = 99.11%, val_loss = 0.0980 &amp; val_accuracy = 97.94%<br>
Number of parameters = 266610</p>
<p>Saving model with lowest val_loss = 0.0980</p>
<p>epoch: 2 training loss = 0.0266, training accuracy = 99.11%, val_loss = 0.0980 &amp; val_accuracy = 97.94%<br>
Number of parameters = 266610</p>
<p>epoch: 3 training loss = 0.0266, training accuracy = 99.11%, val_loss = 0.0980 &amp; val_accuracy = 97.94%<br>
Number of parameters = 266610</p>
<p>epoch: 4 training loss = 0.0266, training accuracy = 99.11%, val_loss = 0.0980 &amp; val_accuracy = 97.94%<br>
Number of parameters = 266610</p>
<p>epoch: 5 training loss = 0.0266, training accuracy = 99.11%, val_loss = 0.0980 &amp; val_accuracy = 97.94%<br>
Number of parameters = 266610</p>
<p>epoch: 6 training loss = 0.0266, training accuracy = 99.11%, val_loss = 0.0980 &amp; val_accuracy = 97.94%<br>
Number of parameters = 266610</p>
<p>‘EarlyStopping’ called!</p>
</blockquote>
<p>This shows that:</p>
<ol>
<li>
<p>The model is “frozen”. It doesn’t learn anything since and val_accuracy and val_loss values stay the same.</p>
</li>
<li>
<p>The number of parameters stays the same and therefore, the define pruning doesn’t seem to be having any effect.</p>
</li>
</ol>
<p>How can I fix them?</p>

ashash: <p>When you use the <code>torch.nn.utils.prune</code> module in PyTorch to prune a neural network, it does not actually remove the parameters from the model. Instead, it masks them as zero with a mask matrix during inference. This can be useful in reducing the model size and computational requirements, but it may not provide a true reduction in the number of parameters.</p>
<p>If you are looking for a tool that can truly remove the pruned parameters from your model, this tool <a href="https://github.com/VainF/Torch-Pruning" class="inline-onebox" rel="noopener nofollow ugc">GitHub - VainF/Torch-Pruning: [CVPR-2023] Towards Any Structural Pruning</a> may be useful.</p>