Extension of convolution layer using CFFI

Ja-Keoung_Koo: <p>I want to extend or make my new convolution layer, using C FFI. Even if I saw this page <a href="https://github.com/pytorch/extension-ffi" rel="nofollow noopener">https://github.com/pytorch/extension-ffi</a> , it is hard to get started.</p>
<p>It seems that I can start from some sources from pytorch/torch/lib/…</p>
<p>Could I just know what sources I have to use or a big picture of getting started?</p>

smth: <p>The extension-ffi example showcases implementing a small function to add two tensors in <a href="https://github.com/pytorch/extension-ffi/blob/master/package/my_package/src/my_lib.c">https://github.com/pytorch/extension-ffi/blob/master/package/my_package/src/my_lib.c</a></p>
<p>What additional information would get you started?</p>
<p>You can look at how existing ops are implemented in C here:<br>
<a href="https://github.com/pytorch/pytorch/tree/master/torch/lib/THNN/generic" class="onebox" target="_blank">https://github.com/pytorch/pytorch/tree/master/torch/lib/THNN/generic</a></p>

apaszke: <p>You could also see how the extension FFI has been used to implement CTC loss wrappers in <a href="https://github.com/SeanNaren/warp-ctc/tree/pytorch_bindings/pytorch_binding">this repo</a>.</p>

Ja-Keoung_Koo: <p>Thanks for your information.</p>
<p>In /THNN/docs I found a description of</p>
<p><code>module = nn.SpatialConvolution(nInputPlane, nOutputPlane, kW, kH, [dW], [dH], [padW], [padH])</code></p>
<p>But it seems that there is no SpatialConvolution in THNN/generic.</p>
<p>I am just curious:</p>
<ol>
<li>
<p>Where can I see the convolution code used in PyTorch?</p>
</li>
<li>
<p>PyTorch doesn’t use any codes in /pytorch/torch/lib, right? For example, there is no dilated convolution in PyTorch docs. So I wonder it.</p>
</li>
</ol>

smth: <ol>
<li>THNN/generic/SpatialConvolutionMM and THNN/generic/SpatialDilatedConvolution</li>
<li>It uses codes there. Dilated convolution is provided as a keyword argument in <code>convXd</code> <a href="http://pytorch.org/docs/nn.html#conv2d">http://pytorch.org/docs/nn.html#conv2d</a>
</li>
</ol>

Zijun_Wei: <p>How can I call the <code>im2col</code> function within my own convolution layer?<br>
In SpatialDilatedConvolution it is called in the following way:</p>
<pre><code>THNN_(col2im)(
  THTensor_(data)(gradColumns),
  nInputPlane, inputHeight, inputWidth, outputHeight, outputWidth,
  kH, kW, padH, padW, dH, dW,
  dilationH, dilationW,
  THTensor_(data)(gradInput_n)
);
</code></pre>
<p>But I’m only considering Float type, so can how can I call this function? Something like THNN_Float_im2col?<br>
Are there any header files I need to import?</p>

tom: <p>Hi,</p>
<p>from <code>grep</code>ing through the pytorch source, I would seem that there is no header with the col2im definition, so you would need to add that somewhere and it might not be considered part of the official api…<br>
In the library, the function is THNN_Floatcol2im<br>
(says <code>objdump -t torch/lib/build/THNN/libTHNN.so.1 | grep col2im</code>).</p>
<p>Best regards</p>
<p>Thomas</p>

Zijun_Wei: <p>Thanks!<br>
So what you mean is that I need to declare it in my code and link to libThNN.so.1 right?<br>
Also can you tell the full definition from the lib file (void THNN_Floatcol2im?, what are the parameters included )?</p>

tom: <p>You can use the one from SpatialFullConvolution.c after substituting <code>real</code>–&gt;<code>float</code> (if it took <code>THTensor_</code> you would need to expand that, too, but it seems to take the <code>float*</code> buffers.</p>

Zijun_Wei: <p>Sorry if I didn’t make it clear.<br>
The function I’m going to use is:</p>
<p><code>THNN_Floatim2col(THFloatTensor_data(input_h_w), 1, inputHeight, inputWidth, kH, kW, padH, padW, dH, dW, dilationH, dilationW, THFloatTensor_data(columns));</code></p>
<p>But when I compile the file, the warning comes out:</p>
<pre><code>warning: implicit declaration of function 'THNN_Float_im2col' is invalid in C99 [-Wimplicit-function-declaration]
            THNN_Float_im2col(THFloatTensor_data(input_h_w), 1, inputHeight, inputWidth, kH, kW, padH, padW, dH, dW, dilationH, dilationW, THFloatTensor_data(columns))
</code></pre>
<p>I wonder this might because I need to declare it first. So the question is,  what is the full declaration for THNN_Float_im2col?</p>
<p>I’m new to c, so if I understand this wrong or there is an other way to solve this problem, please let me know.</p>
<p>Many thanks.</p>

tom: <p>I think the declaration is more</p>
<pre><code class="lang-auto">static void THNN_Floatim2col(const float* data_im, const int channels,
      const int height, const int width, const int kernel_h, const int kernel_w,
      const int pad_h, const int pad_w,
      const int stride_h, const int stride_w,
      const int dilation_h, const int dilation_w,
      float* data_col);
</code></pre>
<p>or so (by literally copying the declaration in the file and substituting).</p>
<p>From the message, you seem to have an extra underscore somewhere.</p>
<p>Best regards</p>
<p>Thomas</p>

Zijun_Wei: <p>Where did you copy this?</p>

tom: <p>Hi!</p>
<p>Sorry, I was outdated by a week… <img src="//discuss.pytorch.org/images/emoji/apple/confused.png?v=5" title=":confused:" class="emoji" alt=":confused:"><br>
The commit seems to remove it. I don’t know whether it will be in the .so on master, but I would have doubts.<br>
But you might be able to get it with pytorch 0.2.</p>
<p>Best regards</p>
<p>Thomas</p>
<aside class="onebox githubcommit">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/commit/ecc7579f446be2b012840c651b7a97f737c12d26#diff-513bcbf41154c24adb96e8ecaef39d2cL5" target="_blank" rel="nofollow noopener">github.com/pytorch/pytorch</a>
  </header>
  <article class="onebox-body">
      <a href="https://github.com/soumith" target="_blank" rel="nofollow noopener">
    <img alt="soumith" src="https://avatars0.githubusercontent.com/u/1310570?v=4" class="thumbnail onebox-avatar" width="90" height="90">
  </a>

<h4>
  <a href="https://github.com/pytorch/pytorch/commit/ecc7579f446be2b012840c651b7a97f737c12d26" target="_blank" rel="nofollow noopener">Merge commit 'e4c05c2b5f3dbc121c0cf4bb78d15540412dcd3c'</a>
</h4>


<div class="date">
  by <a href="https://github.com/soumith" target="_blank" rel="nofollow noopener">soumith</a>
  on <a href="https://github.com/pytorch/pytorch/commit/ecc7579f446be2b012840c651b7a97f737c12d26" target="_blank" rel="nofollow noopener">11:37AM - 25 Aug 17</a>
</div>

<div class="github-commit-stats">
  changed <strong>12 files</strong>
  with <strong>1644 additions</strong>
  and <strong>927 deletions</strong>.
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


Zijun_Wei: <p>I see, thanks.<br>
I tried to add the declaration on top of my C code and got the following warning:</p>
<p><code>THNN_Floatim2col' has internal linkage but is not defined</code></p>
<p>If I removed the declaration I got:</p>
<p><code>implicit declaration of function 'THNN_Floatim2col' is invalid in C99</code></p>
<p>I will continue to check if it is possible to get the header files that defines THNN_Floatim2col.</p>
<p>Thanks for you help.<br>
Best<br>
Z</p>

Zijun_Wei: <p><a class="mention" href="/u/smth">@smth</a> <a class="mention" href="/u/apaszke">@apaszke</a> Guys could you please give some suggestions?</p>

111128: <p>hi<br>
I recently implemented a convolution with the PyTorch extension <a href="https://github.com/NicesMan/PyTorch-extension-Convolution" rel="noopener nofollow ugc">here</a> and the code is based on <a href="https://github.com/pytorch/pytorch/blob/master/aten/src/THCUNN/generic/SpatialConvolutionMM.cu" rel="noopener nofollow ugc">here</a>.<br>
It is worth mentioning that the convolution achieved by PyTorch is very inefficient:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/b/7ba996928753120d8d2e6e96c56e010cd328ed4c.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/7ba996928753120d8d2e6e96c56e010cd328ed4c" title="3_11"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/b/7ba996928753120d8d2e6e96c56e010cd328ed4c_2_690x379.jpeg" alt="3_11" data-base62-sha1="hDY4kvqIkePT8hQKgEJUxv6Mass" width="690" height="379" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/b/7ba996928753120d8d2e6e96c56e010cd328ed4c_2_690x379.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/7/b/7ba996928753120d8d2e6e96c56e010cd328ed4c_2_1035x568.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/7/b/7ba996928753120d8d2e6e96c56e010cd328ed4c.jpeg 2x" data-dominant-color="F1EFD1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">3_11</span><span class="informations">1302×716 182 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
I hope this helps you !</p>

mhubii: <p>thanks for sharing <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:">, but pytorch’s convolutions is way faster than the extension. Do you think it would be possible to speed it up?</p>