Combination of MRI image and its segmentation

maryma: <p>Hi I wanted to train a model for survival time prediction with MRI images. Suppose I have both images and labels and I want to make a dataset that contains both image and label. What can I do for example is there any way to give segmentation as a new channel map? Or is multiplying the tensors of images and labels, a good way? I dont have experience , if someone has please guide me.<br>
My purpose is that my survival prediction model just look at the tumors part instead of whole organ.</p>

KFrank: <p>Hi Maryam!</p>
<aside class="quote no-group" data-username="maryma" data-post="1" data-topic="155425" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/maryma/48/56556_2.png" class="avatar"> maryma:</div>
<blockquote>
<p>is there any way to give segmentation as a new channel map? Or is multiplying the tensors of images and labels, a good way?</p>
</blockquote>
</aside>
<p>You can look at it both ways.</p>
<p>For context, as I understand it, you have an MRI image.  To be concrete, let’s<br>
imagine that this is a single-channel, gray-scale image where the pixel values<br>
run from 0.0 to 1.0.</p>
<p>Then you also have a label (that comes from some external source and is<br>
<em>not</em> what you are trying to train your network to predict).  This label is also an<br>
“image” that is the same size as your image.  To be concrete, lets imagine<br>
that this is also a single-channel, gray-scale image where the pixel values<br>
are 0.0 for pixels in the actual image that are non-tumor pixels and are 1.0<br>
for pixels in the actual image that are tumor pixels.</p>
<p>One the one hand, you could combine your image and label into a single<br>
image by multiplying them together.  Now your non-tumor pixels are masked<br>
out.  That is to say that what were non-tumor pixels are now “background”<br>
pixels with value 0.0.  Your tumor pixels remain whatever they were in your<br>
original image.</p>
<p>The advantage of this approach is that it is easier for the network in that you<br>
are only giving the network the “important” pixels to process.  The network<br>
doesn’t have to “learn” that your label (the zero-one mask image) is telling<br>
it what pixels are important.  If you can “tell” your network something, rather<br>
than making your network “learn” it, training the network is easier.</p>
<p>On the other hand, you could combine your image and label into a two-channel<br>
image by <code>stack()</code>ing them together.  This has the following potential advantage:<br>
I could well believe that the potential malignancy of tumor depends not only on<br>
the tumor tissue itself – the tumor pixels in your image – but on the surrounding<br>
tissue as well – your non-tumor pixels.  Perhaps it matters for survival whether<br>
the tissue surrounding the tumor is dense or vascularized or whatever.</p>
<p>If you mask out (set to zero) the non-tumor pixels, the network won’t have<br>
any information about the non-tumor tissue, so it won’t be able to take the<br>
structure of the non-tumor tissue into account when learning to make its<br>
survival prediction.</p>
<p>So I think that your answer depends on the answer to a biology question.<br>
If the likelihood of survival really only depends on the characteristics of the<br>
tumor tissue, then the approach of using a single-channel input image with<br>
the non-tumor pixels masked out will likely be better (because it makes the<br>
network’s learning task somewhat easier).  But if survival depends in a<br>
significant way also on the non-tumor tissue, you will want the approach<br>
of using a two-channel image – the original image <code>stack()</code>ed with the label<br>
“image” – because that’s how you pass in the non-tumor-tissue information<br>
that the network needs to make the best prediction about survival.</p>
<p>Best.</p>
<p>K. Frank</p>

maryma: <p>Thanks for your complete answer. You are completely right because when I spoke to clinicians they told me the surrounding of the tumor is important but I have some technical issues.<br>
I used monai for making my dataset because they were 3d and handling 3d images is easier with monai. I made a dataset that has the image and the groundtruth. Then I tried to make a custom dataset with pytorch that get image and groundtruth and as you told me I stacked them together.</p>
<pre><code class="lang-auto">def __getitem__(self, index):
            if type(index) is not int:
                raise ValueError(f"Need `index` to be `int`. Got {type(index)}.")
            #extract the image pixels from the dictionary of dataset
            img = self.image_dataset[index]['image']
            lbl = self.image_dataset[index]['label']
            concatim = torch.stack((img,lbl))
            return concatim, (self.time[index], self.event[index])
</code></pre>
<p>Then I made dataloader of this custom dataset:</p>
<pre><code class="lang-auto">def collate_fn(batch):
    """Stacks the entries of a nested tuple"""
    return tt.tuplefy(batch).stack()

dl_train = DataLoader(dataset_train, batch_size=1, shuffle=True, collate_fn=collate_fn)
dl_val = DataLoader(dataset_val, batch_size=1, shuffle=True, collate_fn=collate_fn)
</code></pre>
<p>And in my model I tried to change the input channel of 3d CNN to 2. This is my model:</p>
<pre><code class="lang-auto">    class Net(nn.Module):
        def __init__(self, out_features):
            super().__init__()
            # numbers in front of conv3d are input channel,number of filters,kernel size,stride
            self.conv1 = nn.Conv3d(1, 16, 5, 1)
            self.max_pool = nn.MaxPool3d(2)
            self.conv2 = nn.Conv3d(16, 16, 5, 1)
            self.glob_avg_pool = nn.AdaptiveAvgPool3d((1, 1, 1))
            self.fc1 = nn.Linear(16, 16)
            self.fc2 = nn.Linear(16, out_features)
            self.dropout = nn.Dropout3d(0.25)
            self.dropout1 = nn.Dropout(0.25)
            
        def forward(self, x):
            x = F.relu(self.conv1(x))
            x = self.max_pool(x)
            x = self.dropout(x)
            x = F.relu(self.conv2(x))
            x = self.glob_avg_pool(x)
            x = torch.flatten(x, 1)
            x = F.relu(self.fc1(x))
            x = self.dropout1(x)
            x = self.fc2(x)
            return x
</code></pre>
<p>After fitting my model I faced the error :<br>
RuntimeError: Expected 5-dimensional input for 5-dimensional weight [16, 1, 5, 5, 5], but got 6-dimensional input of size [1, 2, 1, 128, 128, 128] instead<br>
16 is the number of my first CNN filters 5 is the size of filter. 128 is the width, height and depth of my 3d images.</p>

KFrank: <p>Hi Maryam!</p>
<aside class="quote no-group" data-username="maryma" data-post="3" data-topic="155425" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/maryma/48/56556_2.png" class="avatar"> maryma:</div>
<blockquote>
<pre><code class="lang-auto">            concatim = torch.stack((img,lbl))
</code></pre>
</blockquote>
</aside>
<p>It looks like your <code>img</code> and <code>lbl</code> already have an <code>nChannel</code> dimension (with<br>
<code>nChannel = 1</code>), so you will want to use <code>cat()</code>, rather than <code>stack()</code>.  See<br>
below.</p>
<aside class="quote no-group">
<blockquote>
<p>And in my model I tried to change the input channel of 3d CNN to 2.</p>
<pre><code class="lang-auto">            # numbers in front of conv3d are input channel,number of filters,kernel size,stride
            self.conv1 = nn.Conv3d(1, 16, 5, 1)
</code></pre>
</blockquote>
</aside>
<p>This should be:</p>
<pre data-code-wrap="python"><code class="lang-python">            self.conv1 = nn.Conv3d (2, 16, 5, 1)
</code></pre>
<p>(In the code you posted, you have input channels of 1, rather than 2.)</p>
<aside class="quote no-group">
<blockquote>
<p>After fitting my model I faced the error :<br>
RuntimeError: Expected 5-dimensional input for 5-dimensional weight [16, 1, 5, 5, 5], but got 6-dimensional input of size [1, 2, 1, 128, 128, 128] instead</p>
</blockquote>
</aside>
<p>Use <code>cat()</code> rather than <code>stack()</code>.  (<code>stack()</code> adds a new dimension to your<br>
tensor, while <code>cat()</code> uses the <code>nChannel</code> dimension that you already have.)</p>
<p>Consider:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.12.0'
&gt;&gt;&gt; nBatch = 1
&gt;&gt;&gt; nChannel = 1
&gt;&gt;&gt; hwd = 128
&gt;&gt;&gt; img = torch.randn (nBatch, nChannel, hwd, hwd, hwd)
&gt;&gt;&gt; lbl = torch.randn (nBatch, nChannel, hwd, hwd, hwd)
&gt;&gt;&gt; img.shape   # has nBatch = 1, nChannel = 1
torch.Size([1, 1, 128, 128, 128])
&gt;&gt;&gt; concatim = torch.cat ((img, lbl), dim = 1)
&gt;&gt;&gt; concatim.shape   # has nBatch = 1, nChannel = 2
torch.Size([1, 2, 128, 128, 128])
&gt;&gt;&gt; conv = torch.nn.Conv3d (in_channels = 2, out_channels = 16, kernel_size = 5)   # takes 2 input channels
&gt;&gt;&gt; conv (concatim).shape
torch.Size([1, 16, 124, 124, 124])
</code></pre>
<p>Best.</p>
<p>K. Frank</p>

maryma: <p>Why did you choose dim=1 in this part?</p>
<pre><code class="lang-auto">concatim = torch.cat ((img, lbl), dim = 1)
</code></pre>
<p>And because I do not have the number of batches and number of channels in my custom dataset yet ( in my mind they are produced after making dataloader)so should I concat them in dimension 0? I wrote this one<br>
<code>concatim = torch.cat((img, lbl),0)</code><br>
And I got this error<br>
<code>Given groups=1, weight of size [16, 1, 5, 5, 5], expected input[1, 2, 128, 128, 128] to have 1 channels, but got 2 channels instead</code><br>
And I changed the first conv input channel to 2</p>
<pre><code class="lang-auto">    class Net(nn.Module):
        def __init__(self, out_features):
            super().__init__()
            #numbers in front of conv3d are input channel,number of filters,kernel size,stride
            self.conv1 = nn.Conv3d(2, 16, 5, 1)
</code></pre>
<p>And when I tried to contact them in dimension 1 as you wrote here I got this error</p>
<p><code>torch.cat(): Sizes of tensors must match except in dimension 1. Got 115 and 113 in dimension 2 (The offending index is 1)</code></p>
<p>Befor that I tried to change the images and labels dimension to 128 with Monai. You can see it in Resized:</p>
<pre><code class="lang-auto">train_transforms = Compose(
    
        [
            LoadImaged(keys=['image', 'label']),
            #add channels to the images that monai can interpret in this way
            AddChanneld(keys=['image', 'label']),
            #rescale the voxels pixdim=(height,width,depth) 
            Spacingd(keys=['image', 'label'], pixdim=(1.5, 1.5, 2)),
            Orientationd(keys=['image', 'label'], axcodes="RAS"),
            ScaleIntensityRanged(keys='image', a_min=-100, a_max=1000, b_min=0.0, b_max=1.0, clip=True),
            CropForegroundd(keys=['image', 'label'], source_key='image'),
            Resized(keys=['image', 'label'], spatial_size=[128,128,128]),
            #to tensor should be the last transform
            ToTensord(keys=['image', 'label'])
        ]
    )
</code></pre>
<p>Do you think where the problem is?</p>
<p>And I have to add that the output size of img and lbl are</p>
<pre><code class="lang-auto">torch.Size([1, 128, 128, 128])
torch.Size([1, 128, 128, 128])
</code></pre>

KFrank: <p>Hi Maryam!</p>
<aside class="quote no-group" data-username="maryma" data-post="5" data-topic="155425" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/maryma/48/56556_2.png" class="avatar"> maryma:</div>
<blockquote>
<p>Why did you choose dim=1 in this part?</p>
<pre><code class="lang-auto">concatim = torch.cat ((img, lbl), dim = 1)
</code></pre>
</blockquote>
</aside>
<p>I chose <code>dim = 1</code> because I was assuming the <code>img</code> and <code>lbl</code> already<br>
had an <code>nBatch</code> dimension so that the <code>nChannel</code> dimension would be<br>
in position <code>1</code>.</p>
<aside class="quote no-group">
<blockquote>
<p>And because I do not have the number of batches and number of channels in my custom dataset yet ( in my mind they are produced after making dataloader)</p>
</blockquote>
</aside>
<p>Everything you’ve posted suggests that you do have an <code>nChannel</code><br>
dimension (for <code>img</code> and <code>lbl</code>), even if that dimension has size <code>1</code>.</p>
<p>To check this you could print out <code>img.shape</code> and <code>lbl.shape</code> right<br>
before calling <code>concatim = torch.cat((img, lbl),0)</code>.  I expect<br>
that the two <code>shape</code>s will be the same and be <code>[1, 128, 128, 128]</code>.<br>
If this is the case then you do have an <code>nChannel = 1</code> dimension<br>
(and no <code>nBatch</code> dimension yet).</p>
<p>It does make sense that you do not yet have an <code>nBatch</code> and that the<br>
<code>nBatch</code> dimension that appears later is indeed added by the <code>Dataloader</code>.</p>
<aside class="quote no-group">
<blockquote>
<p>so should I concat them in dimension 0? I wrote this one<br>
<code>concatim = torch.cat((img, lbl),0)</code></p>
</blockquote>
</aside>
<p>Yes, if you have an <code>nChannel</code> dimension, but do not have an <code>nBatch</code><br>
dimension, you do want <code>concatim = torch.cat((img, lbl),0)</code>,<br>
that is, with <code>dim = 0</code>.</p>
<aside class="quote no-group">
<blockquote>
<p>And I got this error<br>
<code>Given groups=1, weight of size [16, 1, 5, 5, 5], expected input[1, 2, 128, 128, 128] to have 1 channels, but got 2 channels instead</code><br>
And I changed the first conv input channel to 2</p>
</blockquote>
</aside>
<p>Did you change “the first conv input channel to 2” <em>before</em> you got this<br>
error?  The <code>weight</code> in the error message you quoted only has one input<br>
channel.  Consider:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'1.12.0'
&gt;&gt;&gt; torch.nn.Conv3d (in_channels = 1, out_channels = 16, kernel_size = 5).weight.shape
torch.Size([16, 1, 5, 5, 5])
&gt;&gt;&gt; # versus
&gt;&gt;&gt; torch.nn.Conv3d (in_channels = 2, out_channels = 16, kernel_size = 5).weight.shape
torch.Size([16, 2, 5, 5, 5])
</code></pre>
<aside class="quote no-group">
<blockquote>
<p>And when I tried to contact them in dimension 1 as you wrote here I got this error</p>
</blockquote>
</aside>
<p>As noted above, call <code>cat()</code> with <code>dim = 0</code> (not <code>dim = 1</code>).</p>
<aside class="quote no-group">
<blockquote>
<p><code>torch.cat(): Sizes of tensors must match except in dimension 1. Got 115 and 113 in dimension 2 (The offending index is 1)</code></p>
</blockquote>
</aside>
<p>Regardless of whether you used <code>dim = 1</code> or <code>dim = 0</code> with <code>cat()</code>, this error<br>
message means that you’ve changed something else along the way.</p>
<p>Print out <code>img.shape</code> and <code>lbl.shape</code> right before calling <code>cat()</code>.</p>
<p>Is it possible that the shapes of your images (and labels) change from<br>
iteration to iteration so that sometimes the call to <code>cat()</code> works (when the<br>
shapes match), but then the call to <code>cat()</code> fails in a later iteration when<br>
the shapes don’t match?</p>
<aside class="quote no-group">
<blockquote>
<p>And I have to add that the output size of img and lbl are</p>
<pre><code class="lang-auto">torch.Size([1, 128, 128, 128])
torch.Size([1, 128, 128, 128])
</code></pre>
</blockquote>
</aside>
<p>But in what version of your code do <code>img</code> and <code>lbl</code> have these shapes?<br>
Print them out right before the call to <code>cat()</code> that is giving you the error.</p>
<p>Best.</p>
<p>K. Frank</p>