Conv3D model input tensor

username: gkrisp9
cooked: <p>Hello, I am new to PyTorch and I want to make a classifier for 3D DICOM MRIs. I want to use the pretrained resnet18 from monai library but I am confused with the input dimensions of the tensor. The shape of the images in my dataloader is [2,160,256,256] where 2 is the batch_size, 160 is the number of dicom images for each patient and 256x256 is the dimension of the images. When I try to run the model I get this error: <strong>Expected 5-dimensional input for 5-dimensional weight [64, 3, 7, 7, 7], but got 4-dimensional input of size [2, 160, 256, 256] instead</strong></p>
<p>If I unsqueeze the tensor before feeding it to the model I get: <strong>Given groups=1, weight of size [64, 3, 7, 7, 7], expected input[1, 2, 160, 256, 256] to have 3 channels, but got 2 channels instead</strong> Can anybody help me figure this out ?</p>

username: KFrank
cooked: <p>Hi gkrisp!</p>
<aside class="quote no-group" data-username="gkrisp9" data-post="1" data-topic="139379" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/gkrisp9/48/44914_2.png" class="avatar"> gkrisp9:</div>
<blockquote>
<p><strong>Expected 5-dimensional input for 5-dimensional weight [64, 3, 7, 7, 7]</strong></p>
</blockquote>
</aside>
<p>This is telling you that the first <code>Conv3d</code> layer of your resnet has a<br>
weight with shape <code>[64, 3, 7, 7, 7]</code>, which is to say that is is a<br>
<code>Conv3d (in_channels = 3, out_channels = 64, kernel_size = 7)</code>.</p>
<p>Therefore the input to your resnet (and hence to this <code>Conv3d</code>) must<br>
have shape <code>[nBatch, nChannels = 3, height, width, depth]</code>.</p>
<p>Your first error was caused because your input didn’t have the <code>nBatch</code><br>
dimension.  This dimension can have size one if you want, but it has to<br>
be present.</p>
<p><code>unsqueeze()</code> provided the required <code>nBatch</code> dimension, but you then<br>
get the second error because your input only has 2 channels, while<br>
this particular <code>Conv3d</code> is expecting 3 channels.</p>
<p>Because your resnet is pretrained (and I assume that you don’t want<br>
to throw that away), you should probably add a third channel to your<br>
input, using something redundant like a duplicate of your second channel<br>
or the average of your two “real” channels.</p>
<p>Best.</p>
<p>K. Frank</p>

username: gkrisp9
cooked: <p>Hi Frank, thanks a lot for your answers. I have some more questions if you don’t mind.<br>
Do you think it’s better to use a resnet with 2d convolutions and override the first layer to have input of 160 channels (160 is the number of 256x256 images for each mri) with this command:</p>
<pre><code class="lang-auto">model.features[0] = nn.Conv2d(160,64,7,7,7)
</code></pre>
<p>But in that case, the model takes 160 channels and then squish them to 64. Usually the models first expand the channels before reducing them. Would that be a problem ?</p>
<aside class="quote no-group" data-username="KFrank" data-post="2" data-topic="139379">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>Your first error was caused because your input didn’t have the <code>nBatch</code><br>
dimension. This dimension can have size one if you want, but it has to<br>
be present</p>
</blockquote>
</aside>
<p>The input shape is [2,160,256,256] where theoretically 2 is the nBatch. I don’t know if I’m not getting that right.</p>

username: KFrank
cooked: <p>Hi gkrisp!</p>
<aside class="quote no-group" data-username="gkrisp9" data-post="3" data-topic="139379" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/gkrisp9/48/44914_2.png" class="avatar"> gkrisp9:</div>
<blockquote>
<p>Do you think it’s better to use a resnet with 2d convolutions and override the first layer to have input of 160 channels</p>
</blockquote>
</aside>
<p>It would almost certainly be worse, if I understand correctly that you<br>
are working with 3d images.</p>
<p>The point is that your images have substantive spatial structure.<br>
That is, just as a pixel with <code>x = 17</code> is right next to <code>x = 18</code>, but far<br>
away from <code>x  = 148</code>, a slice in your 3d “z-stack” with <code>z = 17</code> is<br>
right next to the <code>z = 18</code> z-slice, but far away from <code>z = 148</code>.</p>
<p>Convolutions know about and respect this structure, while a general<br>
fully-connected layer does not (and the <code>in_channels</code> → <code>out_channels</code><br>
part of a convolutional layer is fully connected).</p>
<p>You <em>could</em> use a <code>Conv2d</code> with <code>in_channels = 160</code> (and with all of<br>
those extra connections, it would be a superset of the <code>Conv3d</code> so,<br>
in principal, could perform the same inference if properly trained),<br>
but such a network would have to “learn” the spatial structure of your<br>
z dimension, which could take a long time or not happen at all with<br>
realistic amounts of training data and time.</p>
<p>As a general rule, if your problem has some particular structure and<br>
there is a natural way of “telling” your network about that structure<br>
by building that structure into the architecture, you’re much better off<br>
doing so, rather than forcing your network to “learn” that structure.</p>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto">model.features[0] = nn.Conv2d(160,64,7,7,7)
</code></pre>
</blockquote>
</aside>
<p>As an aside, if your model were pretrained, so, in particular,<br>
<code>model.features[0]</code> were pretrained, doing this would throw away<br>
that part of the pretraining, which could be costly.</p>
<aside class="quote no-group">
<blockquote>
<aside class="quote no-group" data-username="KFrank" data-post="2" data-topic="139379">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>Your first error was caused because your input didn’t have the <code>nBatch</code><br>
dimension. This dimension can have size one if you want, but it has to<br>
be present</p>
</blockquote>
</aside>
<p>The input shape is [2,160,256,256] where theoretically 2 is the nBatch.</p>
</blockquote>
</aside>
<p>Yes, my mistake.  You said in your first post that 2 is your batch size.</p>
<p>The basic point still stands, however.  Pytorch’s convolutional layers<br>
require both a batch and a channels dimension, even if they are<br>
“trivial,” <em>singleton</em> (that is, size = 1) dimensions.</p>
<p>So, if your input <code>image</code> has shape <code>[2, 160, 256, 256]</code>, with nBatch = 2,<br>
and no explicit channels dimension, you have to add the required<br>
channel dimension, e.g.:</p>
<pre data-code-wrap="python"><code class="lang-python">image = image.unsqueeze (1)
</code></pre>
<p>However, now you have <code>nChannels = 1</code>, and your pretrained resnet<br>
requires <code>nChannels = 3</code>, so probably the best solution (if you want to<br>
keep the benefits of your pretrained model) is to <code>expand()</code> your input<br>
to have 3 (redundant) channels:</p>
<pre data-code-wrap="python"><code class="lang-python">image = image.unsqueeze (1).expand (-1, 3, -1, -1, 1)
</code></pre>
<p>Now <code>image</code> will have shape <code>[2, 3, 160, 256, 256]</code>.</p>
<p>(Note that <code>expand()</code> just provides a logical view into your 1-channel<br>
image – the 2 additional channels don’t actually “exist” in the sense<br>
of being stored in memory or being able to have values that differ from<br>
those of the one “real” channel.)</p>
<p>Best.</p>
<p>K. Frank</p>

username: gkrisp9
cooked: <pre><code class="lang-auto">image = image.unsqueeze (1).expand (-1, 3, -1, -1, 1)

</code></pre>
<p>I tried this for a small amount of my dataset and it worked. Now I am going to do it on the whole dataset (435 3D MRIs).<br>
So, basically all this does is ‘transforming’ the grayscale images to RGB (3 channels).<br>
Also, I could find a model that initially was trained for grayscale 3D images and therefore I wouldn’t face this problem right ?<br>
Thank you again for sharing your knowledge, you’ve been really helpful!!</p>

username: KFrank
cooked: <p>Hi gkrisp!</p>
<aside class="quote no-group" data-username="gkrisp9" data-post="5" data-topic="139379" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/gkrisp9/48/44914_2.png" class="avatar"> gkrisp9:</div>
<blockquote>
<p>I could find a model that initially was trained for grayscale 3D images and therefore I wouldn’t face this problem right ?</p>
</blockquote>
</aside>
<p>I’ve haven’t experimented with this, so I don’t have any firm knowledge<br>
here.  But based on my intuition, I would say:</p>
<p>In an ideal world, if you have a one-channel, grayscale use case, then<br>
all else being equal, a pretrained one-channel model would seem to<br>
be the best.  But I would expect any one-channel vs. three-channel<br>
benefit to be pretty marginal.  I would think that other details of the<br>
architecture or the quality of pretraining would matter much more.<br>
I just don’t see using a three-channel model on grayscale images as<br>
being a problem in any practical sense.</p>
<p>Best.</p>
<p>K. Frank</p>