Conv2d parameters as a matrix

lsrosa: <p>I have a Conv2d layer, and I need to get a matrix from the layer weights that, when multiplied by the input <code>x</code> will give me the same result as applying the layer <code>x</code>. I have found unfolding-based solutions applied to the input, but in my case, I would like to get the matrix for the Conv2d parameters.</p>
<p>In other words, I need the function to compute the <code>to_matrix()</code> in the code below. I imagine some flattening being necessary for <code>x</code>, which would be fine.</p>
<pre><code class="lang-auto">from torch.nn import Conv2d
n_channels = 2
n_out=3
kernel_size =3
batch_size=2
x_size = (4, 5)

layer = Conv2d(n_channels, n_out, kernel_size)
w = layer.weight
b = layer.bias

x = torch.rand((batch_size, n_channels)+x_size)

y = layer(conv)
my_y = torch.matmul(to_matrix(w, b), x)

print((y-my_y).max())
</code></pre>
<p>Any help  would be appreciated</p>

Aakira: <p>From what I understand, suppose you have a tensor input x with a shape of [C, H, W], and after applying convolution to x, you get a tensor y with a shape of [Cout, Hout, Wout]. At this point, the weight of the conv2D has a shape of <code>[C_out, C*k*k]</code> where k is the kernel size. In this scenario, it’s evident that you would need <code>C_out*H_out*W_out*C_out*C*k*k</code> computations and only <code>C_out*C*k*k</code> parameters.</p>
<p>If you want to vectorize the conv2D operation, it would be similar to applying a dense layer. The number of computations required would be <code>C*H*W*C_out*H_out*W_out</code> computations as well as parameters. I am not sure why you would want to do this, but it clearly has increased the amount significantly.</p>

lsrosa: <aside class="quote no-group" data-username="Aakira" data-post="2" data-topic="209741">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/aakira/48/58984_2.png" class="avatar"> Aakira:</div>
<blockquote>
<p>I am not sure why you would want to do this</p>
</blockquote>
</aside>
<p>Indeed I do not want to perform the <code> torch.matmul(to_matrix(w, b), x)</code> computation, instead, I want to get the matrix for creating a characterization of the matrix created with the layer weights <code> to_matrix(w, b)</code>.</p>

J_Johnson: <p>I’m not sure if this is what you’re looking but it seems related:</p>
<aside class="onebox wikipedia" data-onebox-src="https://en.m.wikipedia.org/wiki/Toeplitz_matrix">
  <header class="source">

      <a href="https://en.m.wikipedia.org/wiki/Toeplitz_matrix" target="_blank" rel="noopener nofollow ugc">en.m.wikipedia.org</a>
  </header>

  <article class="onebox-body">
    

<h3><a href="https://en.m.wikipedia.org/wiki/Toeplitz_matrix" target="_blank" rel="noopener nofollow ugc">Toeplitz matrix</a></h3>

<p>In linear algebra, a Toeplitz matrix or diagonal-constant matrix, named after Otto Toeplitz, is a matrix in which each descending diagonal from left to right is constant. For instance, the following matrix is a Toeplitz matrix:
 Any 
  
    
      
        n
        ×
        n
      
    
    {\displaystyle n\times n}
  
 matrix 
  
    
      
        A
      
    
    {\displaystyle A}
  
 of the form
 is a Toeplitz matrix. If the 
  
    
      
        i
        ,
        j
      
    
    {...</p>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>And this thread might help:</p>
<aside class="quote" data-post="1" data-topic="52968">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mclawrence/48/1496_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/obtaining-toeplitz-matrix-for-convolution/52968">Obtaining Toeplitz matrix for convolution</a> 
  </div>
  <blockquote>
    In this <a href="https://github.com/pytorch/pytorch/issues/6124#issuecomment-404950023" rel="noopener nofollow ugc">issue</a> <a class="mention" href="/u/ezyang">@ezyang</a> references an implementation of convolutions that uses the Toeplitz matrix. 
I have a state_dict (and also a nn.Module class) from a network and explicitly need these Toeplitz matrices for further calculations but I admittedly have not a strong grasp on the things going on in ATen and how I could use that directly in Python. Is there a way to do this?
  </blockquote>
</aside>

<p>By the way, using a Toeplitz matrix may not be the most efficient approach. This post provides a more efficient algorithm with less calculations:</p>
<aside class="onebox stackexchange" data-onebox-src="https://stackoverflow.com/questions/56702873/is-there-an-function-in-pytorch-for-converting-convolutions-to-fully-connected-n/70298690#70298690">
  <header class="source">

      <a href="https://stackoverflow.com/questions/56702873/is-there-an-function-in-pytorch-for-converting-convolutions-to-fully-connected-n/70298690#70298690" target="_blank" rel="noopener nofollow ugc">stackoverflow.com</a>
  </header>

  <article class="onebox-body">
      <a href="https://stackoverflow.com/users/2299067/user2299067" target="_blank" rel="noopener nofollow ugc">
    <img alt="user2299067" src="https://www.gravatar.com/avatar/13bc78777d4ec9a2e7732698154f0a0e?s=256&amp;d=identicon&amp;r=PG" class="thumbnail onebox-avatar" width="256" height="256">
  </a>

<h4>
  <a href="https://stackoverflow.com/questions/56702873/is-there-an-function-in-pytorch-for-converting-convolutions-to-fully-connected-n/70298690#70298690" target="_blank" rel="noopener nofollow ugc">Is there an function in PyTorch for converting convolutions to fully-connected networks form?</a>
</h4>

<div class="tags">
  <strong>neural-network, conv-neural-network, pytorch</strong>
</div>

<div class="date">
  
  answered by
  <a href="https://stackoverflow.com/users/2299067/user2299067" target="_blank" rel="noopener nofollow ugc">
    user2299067
  </a>
  on <a href="https://stackoverflow.com/questions/56702873/is-there-an-function-in-pytorch-for-converting-convolutions-to-fully-connected-n/70298690#70298690" target="_blank" rel="noopener nofollow ugc">01:25AM - 10 Dec 21 UTC</a>
</div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>But still needs the operations to be parallelized.</p>

lsrosa: <p>After some time I pulled out the following function, which computes a sparse matrix from a <code>nn.Conv</code> layer. The implementation could be improved by handling the padding properly, but padding the input images work (see the test script below).</p>
<pre><code class="lang-auto">def c2s(input_shape, weight, bias, stride=(1, 1), padding=(0, 0), dilation=(1,1), device='cpu', verbose=False, warns=True):

    if dilation != (1,1):
        raise RuntimeError('This functions does not account for dilation, if you extendent it, please send us a PR ;).')
    
    if padding != (0, 0):
        input_shape = input_shape[0:1] + torch.Size([x+2*y for x, y in zip(input_shape[-2:], padding)])
        if warns: warn('Do not forget to pad your input accoding to the Conv2d padding. Deactivate this warning passing warns=False as argument.', stacklevel=2)

    Cin, Hin, Win = input_shape
    Cout = weight.shape[0]
    Hk = weight.shape[2]
    Wk = weight.shape[3]
    kernel = weight

    Hout = int(np.floor((Hin - dilation[0]*(Hk - 1) -1)/stride[0] + 1))
    Wout = int(np.floor((Win - dilation[1]*(Wk - 1) -1)/stride[1] + 1))

    shape_out = torch.Size((Cout*Hout*Wout, Cin*Hin*Win+1))
    
    crow = (torch.linspace(0, shape_out[0], shape_out[0]+1)*(Hk*Wk*Cin+1)).int()
    nnz = crow[-1]
    
    # getting columns
    cols = torch.zeros(Cout*Hout*Wout, Cin*Hk*Wk+1, dtype=torch.int)
    data = torch.zeros(Cout*Hout*Wout, Cin*Hk*Wk+1)
    
    base_row = torch.zeros(Cin*Hk*Wk, dtype=torch.int)
    for cin in range(Cin):
        c_shift = cin*(Hin*Win)
        for hk in range(Hk):
            h_shift = hk*Win
            for wk in range(Wk):
                idx = cin*Hk*Wk+hk*Wk+wk
                w_shift = wk
                base_row[idx] = c_shift+h_shift+w_shift
        
    for cout in range(Cout): 
        k = kernel[cout]
        _d = torch.hstack((k.flatten(), bias[cout]))
        for ho in range(Hout):
            h_shift = ho*Win*stride[0]
            for wo in range(Wout):
                w_shift = wo*stride[1]
                idx = cout*Hout*Wout+ho*Wout+wo
                shift = h_shift+w_shift
                cols[idx,:-1] = base_row+shift
                data[idx] = _d

    # add bias as the last column                    
    cols[:,-1] = Cin*Hin*Win

    cols = cols.flatten()
    data = data.flatten()

    csr_mat = torch.sparse_csr_tensor(crow, cols, data, size=shape_out, device=device)
    return csr_mat 
</code></pre>
<p>and to test it:</p>
<pre><code class="lang-auto">import torch
from models.conv2d_to_sparse import conv2d_to_sparse as c2s
from time import time
from numpy.random import randint as ri
from torch.nn.modules.utils import _reverse_repeat_tuple
from torch.nn.functional import pad

if __name__ == '__main__':
    use_cuda = torch.cuda.is_available()
    cuda_index = torch.cuda.device_count() - 2
    device = torch.device(f"cuda:{cuda_index}" if use_cuda else "cpu")

    for i in range(30):
        nc = ri(2, 20) # n channels
        kw = ri(2, 10) # kernel width 
        kh = ri(2, 10) # kernel height
        iw = ri(10, 50) # image width
        ih = ri(10, 50) # image height
        ns = 1 # n samples
        cic = nc # conv in channels
        coc = ri(2, 20) # conv out channels
        sh = ri(2 ,10)
        sw = ri(2, 10)
        ph = ri(2, 10) 
        pw = ri(2, 10) 
        print('\n-------------------------')
        print('cic, coc: ', cic, coc)
        print('kernel h, w: ', kh, kw)
        print('image h, w: ', ih, iw)
        print('stride h, w: ', sh, sw)
        print('padding h, w: ', ph, pw)
        
        c = torch.nn.Conv2d(cic, coc, (kh, kw), stride=(sh, sw), dilation=(1,1), padding=(ph,pw))
        w = c.weight
        b = c.bias
        
        x = torch.rand(ns, nc, ih, iw)
        r = c(x).to(device)

        t0 = time()
        # pad input image
        pad_mode = c.padding_mode if c.padding_mode != 'zeros' else 'constant'
        x_pad = pad(x, pad=_reverse_repeat_tuple(c.padding, 2), mode=pad_mode) 
        
        my_csr = c2s(x[0].shape, w, b, stride=c.stride, padding=c.padding, dilation=c.dilation, device=device)

        #'''
        print('SVDing')
        s, v, d = torch.svd_lowrank(my_csr, q=300)
        #print('v:',  v)
        print('SVDone')
        #'''
        t_curr = time()-t0

        lc = my_csr.to_dense()
        xu = torch.hstack((x_pad.flatten(), torch.ones(1))).to(device)
        ru = lc@xu
        error = torch.norm(r-ru.reshape(r.shape))/torch.norm(r) 
        print('error ru: ', error)
        print('time: ', t_curr) 
        if error &gt; 1.0:
            raise RuntimeError('go debug that conv.')
        print('-------------------------\n')
</code></pre>