The distribution of prediction is way distorted from the labels'

username: Jerry_h
cooked: <p>I am using some CNN layers to do regression task with MSE LOSS FUNCTION. the distribution of label is kind of normal distribution, but after few iterations of training, the prediction distribution gets so weird.<br>
the right side is label distribution and left side is predictions’<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/0/50cd42bc7303f088ef40fb9da0985a12d934e07b.png" data-download-href="https://discuss.pytorch.org/uploads/default/50cd42bc7303f088ef40fb9da0985a12d934e07b" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/5/0/50cd42bc7303f088ef40fb9da0985a12d934e07b.png" alt="image" data-base62-sha1="bwNXVJq5wkx1O9ZvNffzdi8VFHR" width="690" height="320" data-dominant-color="C0C7CD"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1004×466 5.37 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>the model structure</p>
<pre data-code-wrap="bash"><code class="lang-bash">model4(
  (conv_channel1): Sequential(
    (0): Conv2d(3, 16, kernel_size=(2, 2), stride=(1, 1), padding=(2, 2))
    (1): ReLU()
    (2): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
    (3): Conv2d(16, 32, kernel_size=(2, 2), stride=(1, 1), padding=(2, 2))
    (4): ReLU()
    (5): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
  )
  (conv_channel2): Sequential(
    (0): Conv2d(3, 16, kernel_size=(2, 2), stride=(2, 2), padding=(3, 3))
    (1): ReLU()
    (2): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
    (3): Conv2d(16, 32, kernel_size=(2, 2), stride=(2, 2), padding=(3, 3))
    (4): ReLU()
    (5): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
  )
  (conv_channel3): Sequential(
    (0): Conv3d(1, 8, kernel_size=(3, 3, 3), stride=(1, 1, 1), padding=(2, 2, 2))
    (1): ReLU()
    (2): MaxPool3d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
    (3): Conv3d(8, 16, kernel_size=(3, 2, 2), stride=(1, 1, 1), padding=(1, 1, 1))
    (4): ReLU()
    (5): MaxPool3d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
  )
  # the output of above three Conv Layer are flattened and as the input of dense layer
  (dense): Sequential(
    (0): Linear(in_features=864, out_features=128, bias=True)
    (1): ReLU()
    (2): Linear(in_features=128, out_features=64, bias=True)
    (3): ReLU()
    (4): Linear(in_features=64, out_features=1, bias=True)
    (5): Tanh()
  )
)
</code></pre>
<p>any idea is appreciated!</p>

username: alex.veuthey
cooked: <p>It is hard to debug a training session with only the architecture of the network. Could you share the training code?<br>
How does the loss behave during the training session? Is it decreasing properly? Do you normalize your inputs? Just throwing some ideas…</p>

username: Jerry_h
cooked: <p>sure.<br>
I find. there was a mistake when preprocessing the input. I take absolute value of input which makes the x is asymmetric, but after fix it, the distribution still uneven.</p>
<pre data-code-wrap="python"><code class="lang-python">class model4(nn.Module):

    def __init__(self, in_channels=3):
        super(model4, self).__init__()

        self.conv_channel1 = nn.Sequential(
            nn.Conv2d(
                in_channels=in_channels,
                out_channels=16,
                kernel_size=2,
                stride=1,
                padding=2
            ),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2),

            nn.Conv2d(
                in_channels=16,
                out_channels=32,
                kernel_size=2,
                stride=1,
                padding=2
            ),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2)
        )

        self.conv_channel2 = nn.Sequential(
            nn.Conv2d(
                in_channels=in_channels,
                out_channels=16,
                kernel_size=2,
                stride=2,
                padding=3
            ),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2),

            nn.Conv2d(
                in_channels=16,
                out_channels=32,
                kernel_size=2,
                stride=2,
                padding=3
            ),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2),
        )

        self.conv_channel3 = nn.Sequential(
            nn.Conv3d(
                in_channels=1,
                out_channels=8,
                kernel_size=3,
                stride=1,
                padding=2
            ),
            nn.ReLU(),
            nn.MaxPool3d(kernel_size=2),

            nn.Conv3d(
                in_channels=8,
                out_channels=16,
                kernel_size=(3, 2, 2),
                stride=(1, 1, 1),
                padding=(1, 1, 1)
            ),
            nn.ReLU(),
            nn.MaxPool3d(kernel_size=2)
        )

        self.dense = nn.Sequential(
            nn.Linear(864, 128),
            nn.ReLU(),

            nn.Linear(128, 64),
            nn.ReLU(),

            nn.Linear(64, 1),
            nn.Tanh()
        )

    def forward(self, x):
        x = x.view(-1, 3, 4, 16)
        output_channel1 = self.conv_channel1(x)
        output_channel2 = self.conv_channel2(x)
        input_3d = x.unsqueeze(1)
        output_channel3 = self.conv_channel3(input_3d)
        x = torch.cat((output_channel1.view(output_channel1.size(0), -1),
                       output_channel2.view(output_channel2.size(0), -1),
                       output_channel3.view(output_channel3.size(0), -1)), 1)

        output = self.dense(x) * 3
        return output
</code></pre>
<p>and the distribution of input is<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/5/65253dcf4ae8c38e2ceb8cdd92996a532c6e893b.png" data-download-href="https://discuss.pytorch.org/uploads/default/65253dcf4ae8c38e2ceb8cdd92996a532c6e893b" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/5/65253dcf4ae8c38e2ceb8cdd92996a532c6e893b_2_522x500.png" alt="image" data-base62-sha1="eqM0RAHXxSkVgPRCiqFb9zz1JsT" width="522" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/5/65253dcf4ae8c38e2ceb8cdd92996a532c6e893b_2_522x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/6/5/65253dcf4ae8c38e2ceb8cdd92996a532c6e893b_2_783x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/6/5/65253dcf4ae8c38e2ceb8cdd92996a532c6e893b.png 2x" data-dominant-color="8B9096"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">956×915 331 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>by the way, the negative label is quite easy to predict from my experience since the label is the stock return and the short alpha is hard to trade for some restriction which may make it relatively stable.</p>