Extending Col2Im to support 5-D input tensors

vinayakdsci: <p>Hi all,</p>
<p>I recently had to work with PyTorch’s <code>Col2Im</code> and <code>Im2Col</code> functions, and I found that <code>Col2Im</code> currently supports only 3D and 4D input.</p>
<p>I think it would be great if <code>Col2Im</code> could be extended to support 5D input, and in case the maintainers agree with me, I would be really happy to go ahead and open an issue to track progress, and implement 5D support myself.</p>
<p>Any thoughts on this? Would it be alright if I open an issue?</p>
<p>Thanks!</p>

soulitzer: <p>Do you mean to support an additional batch dimension?</p>

vinayakdsci: <p>No, I mean that along with of 2-d output_size and 2-d kernel_size, we support 3-d output_size, and 3-d kernel_size, too.<br>
I am not sure if that would be adding an additional batch dimension?</p>
<p>I apologise if what I am saying is not clear, if you need more clarification, I would be happy to provide an example, too. Thanks!</p>

soulitzer: <p>Gotcha thanks for the clarification. Opening an issue sounds good.</p>
<p>Though apparently there is a library that already implements it here. <a href="https://discuss.pytorch.org/t/manual-implementation-of-unrolled-3d-convolutions/91021/5" class="inline-onebox">Manual Implementation of Unrolled 3D Convolutions - #5 by Youngseok0001</a>. I guess the discussion would be whether want to upstream the implementation there?</p>

vinayakdsci: <p>That is a great way to go forward, but my thought was that as there is already an iterative implementation of <code>Col2Im</code>(that uses for loops), we could extend that itself to support 5D images.</p>
<p>This code in the Onnx Col2Im reference implementation is similar to what I had in mind:</p>
<aside class="onebox githubblob" data-onebox-src="https://github.com/onnx/onnx/blob/4c3ed5e08be75bbe1eeb6818e490b1b6a370183e/onnx/reference/ops/op_col2im.py#L145">
  <header class="source">

      <a href="https://github.com/onnx/onnx/blob/4c3ed5e08be75bbe1eeb6818e490b1b6a370183e/onnx/reference/ops/op_col2im.py#L145" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/onnx/onnx/blob/4c3ed5e08be75bbe1eeb6818e490b1b6a370183e/onnx/reference/ops/op_col2im.py#L145" target="_blank" rel="noopener nofollow ugc">onnx/onnx/blob/4c3ed5e08be75bbe1eeb6818e490b1b6a370183e/onnx/reference/ops/op_col2im.py#L145</a></h4>



    <pre class="onebox"><code class="lang-py">
      <ol class="start lines" start="135" style="counter-reset: li-counter 134 ;">
          <li>        raise ValueError(</li>
          <li>            f"Given n_input_plane={n_input_plane}, X.shape={X.shape}, "</li>
          <li>            f"output_shape={output_shape}, kernel_shape={kernel_shape}, "</li>
          <li>            f"dilations={dilations}, pads={pads}, strides={strides}, "</li>
          <li>            f"expected size of input's dimension 2 to match the calculated number of "</li>
          <li>            f"sliding blocks {n_blocks} = {block_size}, "</li>
          <li>            f"but got input.size(2)={input_length}.",</li>
          <li>        )</li>
          <li></li>
          <li></li>
          <li class="selected">def col2im_naive_implementation(data, image_shape, kernel_shape, dilations, pads, strides):  # type: ignore</li>
          <li>    """Naive implementation for `col2im`."""</li>
          <li>    n_dims = len(pads) // 2</li>
          <li>    new_pads = np.array([(pads[i], pads[i + n_dims]) for i in range(n_dims)])</li>
          <li>    _col2im_shape_check(data, image_shape, kernel_shape, dilations, new_pads, strides)</li>
          <li></li>
          <li>    data_col = data</li>
          <li>    data_im = np.zeros(image_shape, dtype=data.dtype)</li>
          <li></li>
          <li>    dim_col = []</li>
          <li>    for i in range(n_dims):</li>
      </ol>
    </code></pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>The implementation in Felix Dangel’s library(the one that you mentioned) would in my opinion be difficult to integrate in the existing implementation of <code>Col2Im</code>, because of its use of torch functions on whole tensors, instead of the use of indexing.</p>
<p>Your thoughts on this?</p>

vinayakdsci: <p>Hi <a class="mention" href="/u/soulitzer">@soulitzer</a>, I went ahead and opened an issue, and cc’d you there. Here’s the link: <a href="https://github.com/pytorch/pytorch/issues/129564" class="inline-onebox" rel="noopener nofollow ugc">Support for 5-D output image tensors in `Col2Im`. · Issue #129564 · pytorch/pytorch · GitHub</a>.</p>
<p>Thanks!</p>

vinayakdsci: <p>Hi <a class="mention" href="/u/soulitzer">@soulitzer</a>! Just a gentle ping. Could you have a look at the issue when you find time? (Linked above).</p>
<p>I’ll then go ahead and start the implementation once we are able to decide if we would want to upstream the existing implementation or I should extend the current one.</p>
<p>Thanks a lot!</p>