Data type or format of sparse matrix

DanielC: <p>Hi there, I am working on a program which needs to execute a matrix multiplication a vector. The size of the matrix is 2^{18} x 2 ^ {18}, and the size of the vector is, of cause, 2^{18} x 1. Due to memory limitations, I have used the sparse matrix. However, the computational speed is not satisfying, I’d like to know if there is a better option to handle this matrix</p>
<p>some features of my matrix are:</p>
<ol>
<li>The sparsity of the matrix is about 99.5%</li>
<li>It’s a diagonal symmetric matrix.</li>
<li>it contains only 0 and 1.</li>
<li>it can be separated as 512 x 512 blocks, 15757 blocks contains non-zero element (the percentage of non-zero blocks is about 6%)</li>
</ol>
<p>I’m using the COO format right now, obviously it’s not the best choice for this job. I am also consider to use the BSR(or BSC) format. I guess the BSR format will have advantages in terms of the computational speed, but will cause run out of memory.</p>
<p>I wonder if there is a better data type supported by PyTorch for above matrix, especially considering its symmetry and “binary”.</p>
<p>Appreciate for any replies.</p>

KFrank: <p>Hi Daniel!</p>
<aside class="quote no-group" data-username="DanielC" data-post="1" data-topic="167097" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/d/a3d4f5/48.png" class="avatar"> DanielC:</div>
<blockquote>
<p>Due to memory limitations, I have used the sparse matrix. However, the computational speed is not satisfying,</p>
</blockquote>
</aside>
<p>Because your matrix is binary, the indices in your COO-format matrix map,<br>
in effect, positions in your input vector to positions in your result vector.  So<br>
you can use pytorch indexing facilities to effect the matrix-vector multiplication<br>
(without actually multiplying anything).</p>
<p>Here’s such a scheme (that doesn’t take advantage of the symmetry of the<br>
matrix):</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; print (torch.__version__)
1.13.0
&gt;&gt;&gt;
&gt;&gt;&gt; _ = torch.manual_seed (2022)
&gt;&gt;&gt;
&gt;&gt;&gt; n = 16
&gt;&gt;&gt; m = torch.randint (2, (n, n)).float()   # matrix of zeros and ones (not symmetric)
&gt;&gt;&gt; ms = m.to_sparse()                      # sparse (coo) version of m
&gt;&gt;&gt; v = torch.randn (16)
&gt;&gt;&gt;
&gt;&gt;&gt; resultA = m @ v
&gt;&gt;&gt; resultB = ms @ v
&gt;&gt;&gt;
&gt;&gt;&gt; torch.allclose (resultB, resultA)       # verify that sparse-matrix multiplication works
True
&gt;&gt;&gt;
&gt;&gt;&gt; # matrix-vector "multiplication" that uses sparseness and binary nature of ms
&gt;&gt;&gt; resultC = torch.zeros_like (v)
&gt;&gt;&gt; resultC.index_put_ ((ms.indices()[0],), v[ms.indices()[1]], accumulate = True)
tensor([ -7.7993,  -1.3386,  -1.8570,   0.1574,  -5.2830,  -5.6382, -10.0355,
         -2.1667,  -1.6088,  -6.8637,  -6.9376,  -6.2652,  -4.2707,  -2.4600,
         -3.9741,  -1.9541])
&gt;&gt;&gt;
&gt;&gt;&gt; torch.allclose (resultC, resultA)       # verify that indexing approach works
True
</code></pre>
<p>Logically, this seems to me to be about the best you can do (neglecting<br>
the symmetry of the matrix).  However, how well the indexing operations<br>
map to the gpu (or cpu) pipelines will likely be the most important factor<br>
in the actual performance of this scheme.</p>
<aside class="quote no-group">
<blockquote>
<ol start="2">
<li>It’s a diagonal symmetric matrix.</li>
</ol>
</blockquote>
</aside>
<p>What do you mean by “diagonal symmetric?”  Is it the same as what I would<br>
simply call “symmetric?”  Something else?</p>
<aside class="quote no-group">
<blockquote>
<ol start="4">
<li>it can be separated as 512 x 512 blocks, 15757 blocks contains non-zero element (the percentage of non-zero blocks is about 6%)</li>
</ol>
</blockquote>
</aside>
<p>Unless there is further structure to the block structure that you haven’t<br>
mentioned, I don’t see any way to make use of this.  From the numbers<br>
you give, the sparseness of the values within the block matrices is about<br>
the same as the sparseness of the blocks within the big matrix, so the<br>
overall sparseness doesn’t seem to have any real “block” structure to it.</p>
<aside class="quote no-group">
<blockquote>
<p>especially considering its symmetry</p>
</blockquote>
</aside>
<p>It might be possible to get some savings from the matrix’s symmetry, but<br>
at most that would be a factor of two, which might be lost again by the need<br>
to treat diagonal and off-diagonal elements differently.</p>
<p>Best.</p>
<p>K. Frank</p>