Issue with using sparse matrix representation for a pruned model

payal_244: <p>So, I have trained a model and pruned it using unstructured pruning, which has converted some weights to zero. But the inference time hasn’t improved much, since the 0 weights are also participating in the matrix multiplication at the inference. Since autograd doesn’t support sparse matrix calculation, I’ve retrained the pruned model with it’s dense layer and later I converted it to sparse matrix. But when I used it for evaluation it’s giving me following error:</p>
<p>y_preds = model(X_batch)<br>
File “/sys_apps_01/python/python310/lib/python3.10/site-packages/torch/nn/modules/module.py”, line 1501, in _call_impl<br>
return forward_call(*args, **kwargs)<br>
File “/jupyterhubenc/e132293/Pruning/utils/nn_architecture.py”, line 105, in forward<br>
out = self.fc1(x)<br>
File “/sys_apps_01/python/python310/lib/python3.10/site-packages/torch/nn/modules/module.py”, line 1501, in _call_impl<br>
return forward_call(*args, **kwargs)<br>
File “/sys_apps_01/python/python310/lib/python3.10/site-packages/torch/nn/modules/linear.py”, line 114, in forward<br>
return F.linear(input, self.weight, self.bias)<br>
RuntimeError: addmm_sparse_dense: expected either ‘mat1’ to have sparse layout and ‘mat2’ to have strided layout, got ‘mat1’ with layout Strided and ‘mat2’ with layout Sparse</p>
<p>code for the same:<br>
def evaluation( model,test_loader,device,y_test_data):</p>
<pre><code>model.eval()
all_predictions = []

start_time = time.time()

with torch.no_grad():
    for X_batch in test_loader:
        X_batch = X_batch.to(device)
    
        y_preds = model(X_batch)
        
        all_predictions.append(y_preds.view(-1, 2).cpu().numpy())

end_time = time.time()
inference_time = end_time - start_time

print("Inference time:", inference_time, "seconds")
</code></pre>
<p>Is there any way I can fix this or I can remove the zero weights from pruned model in order to reduce the inference time/</p>

Soumya_Kundu: <aside class="quote" data-post="4" data-topic="209074">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/soumya_kundu/48/71716_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/sparse-matrix-computation-in-pytorch/209074/4">Sparse matrix computation in pytorch</a> 
  </div>
  <blockquote>
    Here’s a gist explaining this 

a clear solution on how I can achieve this in pytorch
  </blockquote>
</aside>

<p>Similar thread discussing the same.</p>
<aside class="onebox githubgist" data-onebox-src="https://gist.github.com/aymuos15/10c8348148d3de5954a6b4cc5cfebf2c">
  <header class="source">

      <a href="https://gist.github.com/aymuos15/10c8348148d3de5954a6b4cc5cfebf2c" target="_blank" rel="noopener nofollow ugc">gist.github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://gist.github.com/aymuos15/10c8348148d3de5954a6b4cc5cfebf2c" target="_blank" rel="noopener nofollow ugc">https://gist.github.com/aymuos15/10c8348148d3de5954a6b4cc5cfebf2c</a></h4>

  <h5>Pruning_nuances_in_pytorch.py</h5>
  <pre><code class="Python">import torch
import torch.nn as nn
import torch.nn.utils.prune as prune
import numpy as np
import matplotlib.pyplot as plt

class SimpleCNN(nn.Module):
    def __init__(self, conv1_out=32, conv2_out=64):
        super(SimpleCNN, self).__init__()
        self.conv1 = nn.Conv2d(1, conv1_out, 3, 1)</code></pre>
   This file has been truncated. <a href="https://gist.github.com/aymuos15/10c8348148d3de5954a6b4cc5cfebf2c" target="_blank" rel="noopener nofollow ugc">show original</a>

<p>
</p>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>This link shows how to actually remove the weights via structured pruning.</p>

payal_244: <p>Hi, <a class="mention" href="/u/soumya_kundu">@Soumya_Kundu</a> custom structured pruning will definitely reduce parameters, but is there no any way that I can do the same thing in unstructured pruning also by completely removing zero weights. Is this not achievable with sparse matrix representation?</p>

Soumya_Kundu: <p>Specifically for that, you can have a read of this: <a href="https://github.com/pytorch/pytorch/issues/36214" class="inline-onebox" rel="noopener nofollow ugc">Pruning doesn't affect speed nor memory usage · Issue #36214 · pytorch/pytorch · GitHub</a></p>
<p>TLDR Code:</p>
<pre><code class="lang-auto">import torch
import torch.nn.utils.prune as prune

t = torch.randn(100, 100)
torch.save(t, 'full.pth')

p = prune.L1Unstructured(amount=0.9)
pruned = p.prune(t)
torch.save(pruned, 'pruned.pth')

sparsified = pruned.to_sparse()
torch.save(sparsified, 'sparsified.pth')
</code></pre>
<p>I am not sure as to how you can integrate this into an end to end pipeline.</p>