Regression problem for 3D MRI image

banikr: <p><strong>Problem:</strong><br>
I am trying to predict 512 vectors from MR images…Actual MR image size is (172,220,156). The problem in my work is I can not do patch wise training since each brain volume has correlations with other regions…in terms of electromagnetic fields. So I have to feed the whole image to the network. Since feeding the large size image (almost 6 million voxels) is computationally expensive(generates cuda memory error in my GTx 1080Ti GPU) and for data augmentation, we subsampled the main images into 8 lower resolution images.</p>
<p>For every 8 images, there will be same target of a vector of size (1,512). First 256 are real values and last 256 are imaginary ones. Please take a look at the target below:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/3/3/331c9157b592c592223810cf0ee5a42ba3bfe10d.png" alt="target" data-base62-sha1="7i9yxvl1SG5x9BY0K7lUHYTSmjr" width="480" height="360"></p>
<p>I am using ResNet3D architecture:</p>
<pre><code class="lang-auto">ResNet(
  (strConv1): Conv3d(64, 64, kernel_size=(3, 3, 3), stride=(2, 2, 2), padding=(1, 1, 1))
  (strConv2): Conv3d(128, 128, kernel_size=(3, 3, 3), stride=(2, 2, 2), padding=(1, 1, 1))
  (strConv3): Conv3d(256, 256, kernel_size=(3, 3, 3), stride=(2, 2, 2), padding=(1, 1, 1))
  (conv_block1_32): ConvBlock(
    (conv1): Conv3d(1, 32, kernel_size=(3, 3, 3), stride=(1, 1, 1), padding=(1, 1, 1))
    (bn1): BatchNorm3d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (conv2): Conv3d(32, 32, kernel_size=(3, 3, 3), stride=(1, 1, 1), padding=(1, 1, 1))
    (bn2): BatchNorm3d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  )
  (conv_block32_64): residualUnit(
    (conv1): Conv3d(32, 64, kernel_size=(3, 3, 3), stride=(1, 1, 1), padding=(1, 1, 1))
    (bn1): BatchNorm3d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (conv2): Conv3d(64, 64, kernel_size=(3, 3, 3), stride=(1, 1, 1), padding=(1, 1, 1))
    (bn2): BatchNorm3d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (convX): Conv3d(32, 64, kernel_size=(1, 1, 1), stride=(1, 1, 1))
    (bnX): BatchNorm3d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  )
  (conv_block64_128): residualUnit(
    (conv1): Conv3d(64, 128, kernel_size=(3, 3, 3), stride=(1, 1, 1), padding=(1, 1, 1))
    (bn1): BatchNorm3d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (conv2): Conv3d(128, 128, kernel_size=(3, 3, 3), stride=(1, 1, 1), padding=(1, 1, 1))
    (bn2): BatchNorm3d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (convX): Conv3d(64, 128, kernel_size=(1, 1, 1), stride=(1, 1, 1))
    (bnX): BatchNorm3d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  )
  (conv_block128_256): residualUnit(
    (conv1): Conv3d(128, 256, kernel_size=(3, 3, 3), stride=(1, 1, 1), padding=(1, 1, 1))
    (bn1): BatchNorm3d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (conv2): Conv3d(256, 256, kernel_size=(3, 3, 3), stride=(1, 1, 1), padding=(1, 1, 1))
    (bn2): BatchNorm3d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (convX): Conv3d(128, 256, kernel_size=(1, 1, 1), stride=(1, 1, 1))
    (bnX): BatchNorm3d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  )
  (conv_block256_512): residualUnit(
    (conv1): Conv3d(256, 512, kernel_size=(3, 3, 3), stride=(1, 1, 1), padding=(1, 1, 1))
    (bn1): BatchNorm3d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (conv2): Conv3d(512, 512, kernel_size=(3, 3, 3), stride=(1, 1, 1), padding=(1, 1, 1))
    (bn2): BatchNorm3d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (convX): Conv3d(256, 512, kernel_size=(1, 1, 1), stride=(1, 1, 1))
    (bnX): BatchNorm3d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  )
  (fc1): Linear(in_features=394240, out_features=512, bias=True)
  (fc2): Linear(in_features=512, out_features=512, bias=True)
)
</code></pre>
<p><code>convX</code> layers are residual skip connections. And <code>strConv</code> layers are strided convolutional layers to replace <code>Maxpool</code>. After each <code>Conv+BatchNorm</code> <code>ReLU</code> activation laters were used which do not show up above.</p>
<p>Also using weight initialization as below:</p>
<pre><code class="lang-auto">def weights_init(m):
    if isinstance(m, nn.Conv3d):
        torch.nn.init.xavier_uniform_(m.weight.data, init.calculate_gain('relu'))
        m.bias.data.fill_(0)
        # torch.nn.init.xavier_uniform_(m.bias.data)
    elif isinstance(m, nn.BatchNorm3d):
        m.weight.data.normal_(mean=1.0, std=0.02)
        m.bias.data.fill_(0)
    elif isinstance(m, nn.Linear):
        m.weight.data.normal_(0.0, 0.02)
        m.bias.data.fill_(0)
</code></pre>
<p>I trained the model with 144 subsampled images (86,110,78 size) for both 60 and 100 epochs. Adam optimizer was used. Please see the hyperparams:</p>
<pre><code class="lang-auto">Criterion:MSELoss()
Optimizer:Adam (
Parameter Group 0
    amsgrad: False
    betas: (0.9, 0.999)
    eps: 1e-08
    lr: 0.0001
    weight_decay: 0
)
</code></pre>
<p>The outputs are not so great.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/5/5/55b33de1a14fb9a3f3d79c6b8d833b938a548a0f.png" alt="image" data-base62-sha1="ce8BJv1JHFZlF7wPswvsfRXsx6D" width="480" height="360"></p>
<p>Is there anything I am doing wrong?</p>
<p>Train loss look like this:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/7/4/74f1976bc5b7eb819c324ad94b4fb9b811b9060d.png" alt="image" data-base62-sha1="gGwZCJK18igA3XNSmXdmzzce4KF" width="480" height="360"></p>
<p>Anything I am missing bigtime??</p>

n37jan: <p>Hi <a class="mention" href="/u/banikr">@banikr</a>,</p>
<p>I’m trying to solve a problem that has quite similar structure to yours in different applications. And I’m also having quite similar results like yours… mine is not working okay neither.<br>
I’m digging internet and found this thread. Did you make any progress? Any tips you’d like to share?</p>
<p>Thanks.</p>