Is there a way to realize 4d convolution using the ConvNd function

username: Nan_Meng
cooked: <p>I found  that conv1d, conv2d, conv3d use <code>ConvNd = torch._C._functions.ConvNd</code> for forward passing. I just want to know whether there is an efficient way to use ConvNd for 4dimension convolution, since my inputs are 4-dimension (6 dimension if count batchsize and channel.)</p>
<p>Thanks !!!</p>

username: fmassa
cooked: <p>Not for the moment. We do not yet have <code>n</code>d convolution kernels, for <code>n&gt;3</code>.</p>

username: cbarrick
cooked: <p>What’s the blocking issue?</p>
<p>Since we have the <code>ConvNd</code> function, why can’t we implement a 4d convolution just like the others? Why not n-dimensional?</p>
<p>From the Python side, all the convolution layers ultimately delegate to that <code>ConvNd</code> function. Nothing seems special cased for any particular kernel shape.</p>
<p>I think the C++ side is implemented at <code>torch/csrc/autograd/functions/convolution.{h,cpp}</code>. The forward pass has a special case when n=3, but all it seems to do is reshape the kernel by adding/removing dummy dimensions. I’m not entirely sure why though. Perhaps it could be easily generalized? I’m a bit out of my element with these low-level details, so I may be missing something.</p>

username: smth
cooked: <p>it’s mostly that we haven’t implemented the C bits going for ConvNd in the THNN/THCUNN libraries. (untimately autograd/functions/convolution.cpp dispatches to that or to CuDNN).</p>

username: Edward_Hahn
cooked: <p><a class="mention" href="/u/smth">@smth</a> could you please share if there are any developments on this front?</p>

username: smth
cooked: <p><a class="mention" href="/u/edward_hahn">@Edward_Hahn</a> not really, because the demand for 4d convolution is not high. Your best bet is to create N Conv3D layers to do a Conv4D (there will be some boundary effects / issues, so you will have to pad edges appropriately)</p>

username: rbavery
cooked: <p>Just wanted to chime in and say that I’m interested in this feature, it seems like the natural way to represent time series of multi-channel satellite images.</p>

username: ptrblck
cooked: <p>Wouldn’t this correspond to a 3D convolution, where the input would have the shape <code>[batch_size, in_channels, seq, height, width]</code> or is each “image” a volume?</p>

username: pvjosue
cooked: <p><a href="https://github.com/pvjosue/pytorch_convNd" rel="nofollow noopener">Here</a> you can find a fully functional <strong>convNd</strong>  and nD transposed convolution based on conv3d as mentioned by smth.</p>
<p>But I’m also looking forward to a native 4D convolution, mainly due to speed concerns. I’m starting with the implementation, but indeed seems like a long way to go.</p>

username: Nan_Meng
cooked: <p>Thanks for your work!</p>

username: bia
cooked: <p>I’d also LOVE to see this implemented. In my case, it’s a natural way to represent a time series of 3D dimensional microscopy measurements. Hope the pytorch team will consider implementing it.</p>

username: OverLordGoldDragon
cooked: <p>4D eats mad flops</p>
<p><a href="https://jax.readthedocs.io/en/latest/notebooks/convolutions.html#basic-n-dimensional-convolution" rel="noopener nofollow ugc">JAX</a> seems to have N-dim</p>

username: Jason_Leaver
cooked: <p>I’ve been working on this package…</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://github.com/leaver2000/mesoscaler">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">

      <a href="https://github.com/leaver2000/mesoscaler" target="_blank" rel="noopener nofollow ugc">GitHub</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/344;"><img src="https://opengraph.githubassets.com/950b83078f38ea5153c7fca1245d583258ba5f51df4039076ccbb10aa6dc2552/leaver2000/mesoscaler" class="thumbnail" width="690" height="345"></div>

<h3><a href="https://github.com/leaver2000/mesoscaler" target="_blank" rel="noopener nofollow ugc">GitHub - leaver2000/mesoscaler: Vertical extent scaling</a></h3>

  <p>Vertical extent scaling. Contribute to leaver2000/mesoscaler development by creating an account on GitHub.</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Which resamples 4 dimensional (T, Z, Y, X) atmospheric analysis along an isobaric log scale. Inversely scaling resolution and extent.</p>
<p>I’m currently working with 5 channels</p>
<p>B, C, T, Z, Y, X = 4, 5, 3, 6, 40, 80</p>
<p>I could also make use of this feature</p>

username: Daniel_Mueller
cooked: <p>I created an account just to say, yes this is useful, and it is more useful than you probably realize. N dimensional is the way to go, but 4D would be nice. There are serious time savings in training to be had using this feature.</p>

username: sad_robot
cooked: <p>It might be uncommon due to high compute demands. But 4d is actually special because it is the dimensionality needed to represent full space-time (i.e. 3d video like reality). So I suspect it will be important for scientific applications.</p>

username: Altay
cooked: <p>I can confirm we do need this for scientific data. I work with exploration seismology data collected in dimensionality (n, x, y, z) with occasional multiple channels. Our native domain for the data domain ML would look like (B, C, N, X, Y, Z). We use linear convs to project into patch embeddings for transformer layers. Without the N dimension the Conv3D is enough, but to truly put it on the data domain we need Conv4D.</p>
<p>I will have to try the suggested ConvND library.</p>