Equivalence of two approaches for sparsity

Abhishek_Tyagi: <p>Hi Everyone,<br>
I am trying to understand whether the following two ways to achieve a sparse matrix are equivalent in Pytorch. Let me add some context:</p>
<p>I am training Sparse Neural Networks with a specific structured sparsity pattern. To do so, there are two methods that I have looked at.</p>
<ol>
<li>
<p>Parameterized Approach<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/b/f/bfb60e528291fc45d9452eb44f579e39d7172e7d.png" alt="Screenshot 2024-07-13 at 4.42.19 PM" data-base62-sha1="rlXfuHt5TIYp6J5MNYoHoSGunff" width="278" height="145"><br>
In this approach <strong>P_j</strong> is a permutation matrix having 1s in places where the weights are supposed to be non-zero in the final <strong>W</strong> matrix. <strong>P</strong> is pre-defined and is not trainable. Moreover, there is no overlap between two <strong>P</strong> matrices. Also, each <strong>P</strong> has 1s in <strong>D</strong> positions.<br>
<strong>alpha</strong> is a vector of size <strong>J</strong> and can have values as either 1 or 0. It dictates which permutation matrices to choose for the final <strong>W</strong> matrix.</p>
</li>
<li>
<p>Masking Approach<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/7/3/730d8bbcfbce9d7a1661e27ce69fd5f45adedfb6.png" alt="Screenshot 2024-07-14 at 9.43.11 AM" data-base62-sha1="gpNVIB1isiH4iSnwJxqdY8kkbH0" width="187" height="130"><br>
This approach also uses <strong>P</strong> matrices to decide the non-zeros’ position in the final W matrix.<br>
<strong>M</strong> is the final mask after adding <strong>K</strong> permutation matrices.</p>
</li>
</ol>
<p><strong>Comparison</strong><br>
To compare the two methods, we ensure that the resulting <strong>W</strong> has the same number of non-zeros.  Let’s take the example of K = 30.</p>
<ol>
<li>In the first approach, we randomly pick 30 <strong>alphas</strong> to be 1, and the rest all are zero. This will give us a <strong>W</strong> matrix with 30 x D non-zeros.</li>
<li>For the second approach, we use the indices of randomly picked <strong>alphas</strong> and pick the corresponding <strong>P</strong> matrices to form <strong>M</strong></li>
<li>I also set the random seed to be the same for both experiments to initialize <strong>W</strong>.</li>
</ol>
<p>I am using these weight formulations for an MLP on CIFAR-10. My MLP has the following specifications:</p>
<ol>
<li>Layer1: 3072x3072 Weight Matrix</li>
<li>Layer2: 10x3072 Weight Matrix</li>
</ol>
<p>In both experiments, I am sparsifying Layer1 and keeping Layer2 dense.</p>
<p>The dense accuracy of the network is 57.34 % (Not great, but this network is just a toy example for our sparsity research)</p>
<p><strong>When I choose, K =30 and do the experiments, I get an accuracy of around 50.1% with parameterized approach and 56.4% with masking approach.</strong></p>
<p>Does anyone have any suggestions on why that might be the case?<br>
Are the two approaches equivalent in terms of Pytorch implementation of gradient flow?<br>
Any suggestions on how I can debug this?</p>

Abhishek_Tyagi: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> I will appreciate any input on this. Thank you!</p>