Feeding 3D volumes to Conv3D

username: Rhuax
cooked: <p>Hi there, I’m trying to feed 3D volumes through a NN which has a Conv3D layer.<br>
For simplicity I have:</p>
<blockquote>
<p>self.conv1 = nn.Conv3d(125, 2, 3)</p>
</blockquote>
<p>and in the forward:</p>
<blockquote>
<p>return self.conv1(x)</p>
</blockquote>
<p>My volume is 125x256x256. When I try to feed a random tensor:</p>
<blockquote>
<p>n=Net()<br>
i=torch.randn(1,125,256,256)<br>
o=n(i)</p>
</blockquote>
<p>I get:</p>
<blockquote>
<p>RuntimeError: Expected 5-dimensional input for 5-dimensional weight [2, 125, 3, 3, 3], but got input of size [1, 125, 256, 256] instead</p>
</blockquote>
<p>I do not understand why is it asking a 5-dimensional input</p>

username: ptrblck
cooked: <p>If your input has the shape <code>[batch_size, channels, height, width]</code>, you should use a <code>nn.Conv2d</code> layer.<br>
For <code>nn.Conv3d</code> you need an additional dimension, e.g. time or slices through an MRI.</p>

username: Rhuax
cooked: <p>I’m using MRI, but the volume is 256x256x125. Should I use conv2d or conv3d?</p>

username: ptrblck
cooked: <p>I’m not sure what would work best for your use case, but let’s have a look at the differences between both approaches.</p>
<p>I assume your MRI data has a spatial size of <code>256x256</code> and contains <code>125</code> slices.<br>
If you’re using <code>nn.Conv2d</code> I would suggest to use the slices as the “channels”.<br>
This would mean that each kernel in your conv layer will have the defined spatial size, e.g. <code>3</code>, and will use all channels. The kernel shape would be <code>[nb_kernels, 125, 3, 3]</code>. The output will thus be calculated by using a dot product of the small <code>3x3</code> window and <strong>all</strong> slices.</p>
<p>On the other hand, if you are using <code>nn.Conv3d</code>, you could permute the slices to the “depth” dimension and add a single dimension for the channels. This would mean your kernel is now a volume with the shape e.g. <code>[3x3x3]</code>, such that 3 neighboring slices will be used to produce the output for the current position.</p>
<p>What would you like your model to learn in the MRI images? I think in a segmentation case the second approach could work better, but that’s just my guess.</p>

username: Rhuax
cooked: <p>Thank you very much for the explanation, super clear! Right now I’m working on classification, so maybe the first approach is more suitable as well as more light in term of number of parameters.</p>

username: anksng
cooked: <p>Hello guys,<br>
I want to use conv3d over a spatial matrix of words. Each cell represents an embedding vector of size 100. Hence, an input is of shape - [batch size, height , width, embedding_dim].<br>
1 - Should I use in_channels = embedding_dim  ?<br>
2 - I want to get output of size - [batch_size, height, width, output_size], where output_size is any desired int. I wish to get feature of output_size after convolution for each cell to get the spatial feature for a word (cell).<br>
How would I use conv3d in this case? Please suggest the correct usage and if I am thinking correctly.</p>
<p>Thanks</p>

username: ayalaa2
cooked: <p>It depends.</p>
<p>In the Conv2D case, the expected input is <code>[batch_size, in_channels, height, width]</code>. When we perform a 2d convolution, each filter is acting upon <strong>all the channels</strong>. We sort of assume there isn’t a <em>depth</em> aspect in our channels. For example, consider RGB, it’s not like it has to be in that order, we just keep it in that order for convention.</p>
<p>In the Conv3D case, the expected input is <code>[batch_size, in_channels, depth, height, width]</code>. As an example of this would be video. In this case, order does matter since it’s not like we can shuffle all the frames, we end up losing meaning. We use a 3D convolution to embed that information, where each filter strides across our frames (rather than act on them all).</p>
<p>Now for your specific case, it doesn’t sound like you have a “depth” aspect to your data. The entire embedding vector as a whole seems important, it doesn’t make sense to me to stride over your embedding dimension. If I’m understanding correctly, I think you want a Conv2d with <code>kernel_size=1</code> and <code>stride=1</code>. Your <code>in_channels</code> would be <code>embedding_dim</code>, and your <code>output_channels</code> would be whatever feature size. That’ll result in an output of shape <code>[batch_size, output_channels, height, width]</code>. Keep in mind the order of things.</p>

username: ivan_biji
cooked: <p>I have input CT of size 100x512x512, I want to enhance the quality of the CT using corresponding higher quality image. Can I use a Conv3d to stride stride over 3d volumes of neighboring slices to produce a 2d image at the end ?</p>

username: ptrblck
cooked: <p>The 3D convolution would return an output volume, but you could try to reduce one of the dimensions (e.g. the depth).<br>
I.e. an input of <code>[batch_size, channels, depth, height, width]</code> would result in an output of <code>[batch_size, out_channels, depth*, height*, width*]</code>, where the <code>*</code> shapes are calculated depending on the kernel size, stride, dilation etc.</p>

username: Hassan_Imani
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="9" data-topic="32378">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>[batch_size, channels, depth, height, width]</p>
</blockquote>
</aside>
<p>I have a question abou using video with Conv3D.<br>
My input is like [batch_size=128, channels=3, depth=32, height=64, width=64]</p>
<p>Does it matter to feed the input like:<br>
first case: [batch_size=128, channels=3, depth=32, height=64, width=64]<br>
or<br>
second case: [batch_size=128, channels=3, height=64, width=64, depth=32]</p>
<p>If I feed like second case, what will happen? The network will not learn or it will not learn efficiently?</p>
<p>Thank you</p>

username: ptrblck
cooked: <p>I don’t think you should see a significant difference as long as the data augmentation and transformations are applied to the corresponding dimensions, since the activation volume would just be permuted, i.e. the same would happen if you transpose the height and width of an image.</p>

username: Hassan_Imani
cooked: <p>Thank you for the reply.<br>
Let me ask a queston related to conv3d which is related to this post.</p>
<p>I have a code in keras. I have a problem understanding the “same” padding.<br>
My input shape to this layer is:  [128, 2048, 4, 2, 2]<br>
This is my keras code:<br>
combined = Conv3D(128, (3, 3, 3),strides=1,  padding=‘same’)(combined)</p>
<p>And I want to do it in pytorch. What I did:<br>
self.conv1 = nn.Conv3d(2048, 128, kernel_size=(3,3,3), stride=1,  padding=( 1,1,1))</p>
<p>Are these perform the same?</p>

username: ptrblck
cooked: <p>Yes, for a kernel size of <code>(3, 3, 3)</code> (and a stride of <code>1</code>) you would have to use a padding value of <code>(1, 1, 1)</code> to get the same output shape.</p>