Network predicts the same curve independent of data-sample, why?

username: Capo_Mestre
cooked: <p>I define my Dataset class:</p>
<pre><code class="lang-auto">class MyDataset(Dataset):
    def __init__(self, images, n, labels=None, transforms=None):
        self.X = images
        self.y = labels
        self.n = n
        self.transforms = transforms
         
    def __len__(self):
        return (len(self.X))
    
    def __getitem__(self, i):
        data = self.X.iloc[i, :]
#         print(data.shape)
        data = np.asarray(data).astype(np.float).reshape(1,n,n)
        
        if self.transforms:
            data = self.transforms(data).reshape(1,n,n)
            
        if self.y is not None:
            y = self.y.iloc[i,:]
#             y = np.asarray(y).astype(np.float).reshape(2*n+1,) # for 257-vector of labels
            y = np.asarray(y).astype(np.float).reshape(128,) # for 128-vector of labels
            return (data, y)
        else:
            return data
</code></pre>
<p>Then I create the instances of the train, dev, and test data:</p>
<pre><code class="lang-auto">train_data = MyDataset(train_images, n, train_labels, None)
dev_data = MyDataset(dev_images, n, dev_labels, None)
test_data = MyDataset(test_images, n, test_labels, None)
</code></pre>
<p>After having trained my network, I want to see how well it behaves with the <code>test_data</code>. This is what I do to check this.</p>
<pre><code class="lang-auto">network = Network(128).double()

btch_sz = 100 #  batch size (e.g. 100)
loader_test = DataLoader(test_data_normal,batch_size=btch_sz,shuffle=True,num_workers=0)

for data in loader_test:
    images_test, labels_test = data
    labels_test = labels_test.detach().numpy()
    # to get the data samples from the first batch I break the for-loop cycle
    break

outputs_test = network(images_test)
outputs_test = outputs_test.detach().numpy()
</code></pre>
<p>Then I plot comparative graphs comparing predicted <code>outputs_test</code> with ground-truth <code>labels_test</code>:</p>
<pre><code class="lang-auto">nnn = 3 # to chose a specific sample from the first batch of the test dataset (e.g. 3rd data sample in the batch of 100 samples)

figure, ax1 = plt.subplots()
ax1.plot(labels_test[nnn-1,:], color=u'#1f77b4')
ax1.plot(160*outputs_test[nnn-1,:], color=u'#ff7f0e'')
ax1.legend(['True','Predicted'])
ax1.set_ylim([-40,40])
</code></pre>
<p>As you can see, I have to multiply the <code>outputs_test</code> by 160 to make the resulting curve comparable in amplitude with the ground-truth curve. <strong>This is the first question, why would it be so?</strong></p>
<p>Here are the plots for different data-samples from the batch of 100 and for different epochs of the training (the rows represents indicated epochs and the columns represent indicated data-samples from the batch):</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/7/1706beb1ea4e883b15bd0a46a1ab1ce5eead1c61.png" data-download-href="https://discuss.pytorch.org/uploads/default/1706beb1ea4e883b15bd0a46a1ab1ce5eead1c61" title="Untitled"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/7/1706beb1ea4e883b15bd0a46a1ab1ce5eead1c61_2_562x500.png" alt="Untitled" data-base62-sha1="3hHqPk92gPgBO6BD4tYSR0C62Vr" width="562" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/7/1706beb1ea4e883b15bd0a46a1ab1ce5eead1c61_2_562x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/1/7/1706beb1ea4e883b15bd0a46a1ab1ce5eead1c61_2_843x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/1/7/1706beb1ea4e883b15bd0a46a1ab1ce5eead1c61_2_1124x1000.png 2x" data-dominant-color="F9F9F8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Untitled</span><span class="informations">1239×1101 93 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>As you can see, the result doesnt depend much on what data-sample is fed into the network. This is another question, <strong>why is that so</strong>?</p>
<p><strong>Addon:</strong> For the reference this is how cost evolved over 150 epochs.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/4/a/4a5d50f769f8e7d7122b2247292a7f487de0d452.png" alt="Untitled1" data-base62-sha1="aBRgfawxkQ1TW7qmnHB4d2c11U6" width="414" height="235"></p>
<p>Also, my network consists of 10 convolutional layers (including batchnorms and maxpools in some places) and 5 linear fully-connected layers.</p>
<p>This is the output of the <code>print(network)</code>:</p>
<pre><code class="lang-auto">Network(
  (encoder): MyEncoder(
    (conv_blocks): Sequential(
      (0): Sequential(
        (0): Conv2d(1, 6, kernel_size=(5, 5), stride=(1, 1))
        (1): ReLU()
      )
      (1): Sequential(
        (0): Conv2d(6, 12, kernel_size=(5, 5), stride=(1, 1))
        (1): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
        (2): BatchNorm2d(12, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (3): ReLU()
      )
      (2): Sequential(
        (0): Conv2d(12, 32, kernel_size=(5, 5), stride=(1, 1))
        (1): ReLU()
      )
      (3): Sequential(
        (0): Conv2d(32, 32, kernel_size=(5, 5), stride=(1, 1))
        (1): ReLU()
      )
      (4): Sequential(
        (0): Conv2d(32, 16, kernel_size=(5, 5), stride=(1, 1))
        (1): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
        (2): BatchNorm2d(16, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (3): ReLU()
      )
      (5): Sequential(
        (0): Conv2d(16, 16, kernel_size=(5, 5), stride=(1, 1))
        (1): ReLU()
      )
      (6): Sequential(
        (0): Conv2d(16, 14, kernel_size=(5, 5), stride=(1, 1))
        (1): ReLU()
      )
      (7): Sequential(
        (0): Conv2d(14, 14, kernel_size=(5, 5), stride=(1, 1))
        (1): BatchNorm2d(14, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (2): ReLU()
      )
      (8): Sequential(
        (0): Conv2d(14, 12, kernel_size=(5, 5), stride=(1, 1))
        (1): ReLU()
      )
      (9): Sequential(
        (0): Conv2d(12, 12, kernel_size=(5, 5), stride=(1, 1))
        (1): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
        (2): BatchNorm2d(12, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (3): ReLU()
      )
    )
  )
  (decoder): MyDecoder(
    (fc_blocks): Sequential(
      (0): Sequential(
        (0): Linear(in_features=48, out_features=120, bias=True)
        (1): ReLU()
      )
      (1): Sequential(
        (0): Linear(in_features=120, out_features=60, bias=True)
        (1): ReLU()
      )
      (2): Sequential(
        (0): Linear(in_features=60, out_features=40, bias=True)
        (1): BatchNorm1d(40, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (2): ReLU()
      )
      (3): Sequential(
        (0): Linear(in_features=40, out_features=20, bias=True)
        (1): ReLU()
      )
      (4): Sequential(
        (0): Linear(in_features=20, out_features=10, bias=True)
        (1): ReLU()
      )
    )
  )
  (last): Linear(in_features=10, out_features=128, bias=True)
)
</code></pre>

username: ptrblck
cooked: <p>Your model doesn’t seem to learn the underlying distribution properly.<br>
Try to scale down the problem by using a tiny dataset (e.g. just 10 sample) and make sure your model is able to overfit it by playing around with the architecture and hyperparameters.<br>
Once this is done, you could scale up the use case again.</p>

username: Capo_Mestre
cooked: <p>Thanks a lot for this suggestion, <a class="mention" href="/u/ptrblck">@ptrblck</a>! I will try it!</p>

username: Capo_Mestre
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> I tried your suggestion and I could achieve some sensitivity to the input data (I used the <em><strong>batch size of 1</strong></em> and 1000 epochs). Also, the training data was fitted better than the test data, so it looks like overfitting. Although, the fitting of the <em><strong>training data</strong></em> is also not perfect as can be seen in the picture below.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/a/4a6f317b8b0f510c0db3d1c9bdb121365dbeac24.png" data-download-href="https://discuss.pytorch.org/uploads/default/4a6f317b8b0f510c0db3d1c9bdb121365dbeac24" title="ovefrfitting"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/a/4a6f317b8b0f510c0db3d1c9bdb121365dbeac24_2_690x147.png" alt="ovefrfitting" data-base62-sha1="aCtyVu044NMGKdjuxCHtVKPI2KE" width="690" height="147" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/a/4a6f317b8b0f510c0db3d1c9bdb121365dbeac24_2_690x147.png, https://discuss.pytorch.org/uploads/default/optimized/3X/4/a/4a6f317b8b0f510c0db3d1c9bdb121365dbeac24_2_1035x220.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/4/a/4a6f317b8b0f510c0db3d1c9bdb121365dbeac24.png 2x" data-dominant-color="FAF8F6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">ovefrfitting</span><span class="informations">1049×225 30.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>And here is the prediction result for <em><strong>the two test data samples</strong></em> that my network has never seen before:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/d/4/d4cc80e4b8a6469c5b9d613b339ca516b32c5594.png" alt="ovefrfitting_testset" data-base62-sha1="umvmXYF0SguGPejQtikhNXLbOZe" width="516" height="170"></p>
<p>I also <em><strong>tried the batch size of 10</strong></em>, and after 1000 epochs, the result is worse for the <em><strong>training data</strong></em> (compared to the batch size of 1):<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/9/8972ce823e0a1702cadcfc168cbc0952bd516db6.png" data-download-href="https://discuss.pytorch.org/uploads/default/8972ce823e0a1702cadcfc168cbc0952bd516db6" title="ovefrfitting_testset_10"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/9/8972ce823e0a1702cadcfc168cbc0952bd516db6_2_690x151.png" alt="ovefrfitting_testset_10" data-base62-sha1="jBVnUjysBSJaRGTQReIQ0sSPIvY" width="690" height="151" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/9/8972ce823e0a1702cadcfc168cbc0952bd516db6_2_690x151.png, https://discuss.pytorch.org/uploads/default/original/3X/8/9/8972ce823e0a1702cadcfc168cbc0952bd516db6.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/8/9/8972ce823e0a1702cadcfc168cbc0952bd516db6.png 2x" data-dominant-color="F9F0E9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">ovefrfitting_testset_10</span><span class="informations">1027×225 56.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>For the same network architecture I tried the same images, but different “labels” (batch size of 1, 1000 epochs). The labels are <em><strong>oscillatory curves</strong></em> centered around zero vertically and horizontally. The result is not good at all: the prediction is the same for all 10 training data.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/f/0f48a8276b2a208b9bb0149b96da9cf0dbc34dde.png" data-download-href="https://discuss.pytorch.org/uploads/default/0f48a8276b2a208b9bb0149b96da9cf0dbc34dde" title="ovefrfitting_testset_10_oscillator"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/f/0f48a8276b2a208b9bb0149b96da9cf0dbc34dde_2_690x151.png" alt="ovefrfitting_testset_10_oscillator" data-base62-sha1="2bcPniQX8KkCK30NxwvYxbQMqo6" width="690" height="151" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/f/0f48a8276b2a208b9bb0149b96da9cf0dbc34dde_2_690x151.png, https://discuss.pytorch.org/uploads/default/original/3X/0/f/0f48a8276b2a208b9bb0149b96da9cf0dbc34dde.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/0/f/0f48a8276b2a208b9bb0149b96da9cf0dbc34dde.png 2x" data-dominant-color="F8F8F7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">ovefrfitting_testset_10_oscillator</span><span class="informations">1027×225 30.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p><strong>Question: Would you have an idea why oscillatory behaviour is so different?</strong></p>