Neural network could not learn on this type of data, why?

Biplob_Das: <p><strong>Unable to learn by network this type of data</strong></p>
<pre><code class="lang-auto">input_data  = [x for x in range(1, 4000)]
label_data  = [((d**3)) for d in input_data] # using power
</code></pre>
<p><strong>But, This data learns quickly by the same network</strong></p>
<pre><code class="lang-auto">input_data = [x for x in range(1, 4000)]
label_data = [((d*3)) for d in input_data] # using simple multiplication
</code></pre>
<p>My code is:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.utils.data.dataloader as dataloader

import matplotlib.pyplot as plt

# network
class Network(nn.Module):
    def __init__(self):
        super(Network, self).__init__()
        self.fc1 = nn.Linear(in_features=1, out_features=50, bias=True)
        self.fc2 = nn.Linear(in_features=50, out_features=1, bias=True)
    
    def forward(self, x):
        x = F.relu(self.fc1(x))
        x = self.fc2(x)
        return x
        
net = Network()

class MyData(torch.utils.data.Dataset):
    def __init__(self, data_input, data_output):
        self.data_input = data_input
        self.data_output = data_output
        
    def __len__(self):
        return len(self.data_input)
    
    def __getitem__(self, idx):
        return {
            'x':torch.tensor([self.data_input[idx]], dtype=torch.float32), 
            'y':torch.tensor([self.data_output[idx]], dtype= torch.float32)
        }

optimizer = torch.optim.Adam(net.parameters(), lr=0.01)

input_data = [x for x in range(1, 4000)]
# label_data = [((d*3)) for d in input_data] # simple multiplication
label_data  = [((d**3)) for d in input_data]  # using power

batch_size = 100
data = torch.utils.data.DataLoader(dataset=MyData(input_data, label_data), shuffle=True, batch_size=batch_size)
criterion = nn.MSELoss()
x_data, y_data = list(),list()

for e in range(600):
    total_loss = 0
    for i, value in enumerate(data):   
        prediction = net(value['x'])
        loss = criterion(prediction, value['y'])
        total_loss+= loss.item()/batch_size
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
        
    y_data.append(total_loss/i+1)
    x_data.append(e)
        

net(torch.tensor([2], dtype=torch.float32)).item()

plt.figure(figsize=(16,9))
plt.xlabel('Epoch --&gt;', size=20)
plt.ylabel('Loss --&gt;', size=20)
plt.xticks(size=20)
plt.yticks(size=20)
plt.title('Loss graph')

plt.plot(x_data, y_data, color='green', linewidth=2)
plt.show()

print(total_loss/i+1) # print the last loss only
</code></pre>

ptrblck: <p>The range of the target values is completely different (max of <code>63952011999</code> vs. <code>11997</code>) which would explain the failure in learning.<br>
Note that the parameters of a neural network are usually initialized with “small” values using a scaled normal or uniform distribution.<br>
To map the input value of e.g. 3999 to 63952011999 would thus require large weight values, which might take a while during the classical SGD.</p>

Biplob_Das: <p>Thanks <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:">  , i understand it now, <strong>eg: 3999 to 63952011999 would require large weight values</strong>. But neural network weights are initialized with small values. Here i can convert large target values to  smaller range of values (eg: between 0 and 1) to solve it. Will it work then? <img src="https://discuss.pytorch.org/images/emoji/apple/relaxed.png?v=9" title=":relaxed:" class="emoji" alt=":relaxed:"></p>

ZeroMax: <p>I don’t think so, because you have to consider that floating point numbers are really quite limited in what they can express, in the case of a simple sigmoid over large values… If you’re only doing small values, it may generalize to large values, but I think this is highly unlikely…</p>

ptrblck: <aside class="quote no-group" data-username="Biplob_Das" data-post="3" data-topic="96648">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/biplob_das/48/23083_2.png" class="avatar"> Biplob_Das:</div>
<blockquote>
<p>Here i can convert large target values to smaller range of values (eg: between 0 and 1) to solve it. Will it work then? <img src="https://discuss.pytorch.org/images/emoji/apple/relaxed.png?v=12" title=":relaxed:" class="emoji" alt=":relaxed:" loading="lazy" width="20" height="20"></p>
</blockquote>
</aside>
<p>Yes, this could work and is sometimes used if the target doesn’t have “optimal” values.<br>
You could normalize the input (which is usually done anyway) and also normalize the target to train the model.<br>
After the training you could unnormalize the predictions using the target stats and calculate the “real loss” based on this new prediction tensor.</p>
<p>Here is a code snippet using your initial dataset, which can successfully learn the data using this approach:</p>
<pre data-code-wrap="python"><code class="lang-python">class Network(nn.Module):
    def __init__(self):
        super(Network, self).__init__()
        self.fc1 = nn.Linear(in_features=1, out_features=50, bias=True)
        self.fc2 = nn.Linear(in_features=50, out_features=1, bias=True)
    
    def forward(self, x):
        x = F.relu(self.fc1(x))
        x = self.fc2(x)
        return x
        
model = Network()
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)
criterion = nn.MSELoss()

input  = torch.tensor([x for x in range(1, 4000)]).float()
target  = torch.tensor([((d**3)) for d in input]).float()
input = input.unsqueeze(1)
target = target.unsqueeze(1)

# normalize input
input_mean = torch.mean(input)
input_std = torch.std(input)
input = input - input_mean
input = input / input_std

# normalize target
target_mean = torch.mean(target)
target_std = torch.std(target)
target_norm = target - target_mean
target_norm = target_norm / target_std

for epoch in range(1000):
    optimizer.zero_grad()
    out = model(input)
    loss = criterion(out, target_norm)
    loss.backward()
    optimizer.step()
    print('Epoch {}, loss {}, max. pred val {}'.format(
        epoch, loss.item(), out.max()))
        
with torch.no_grad():
    pred = model(input)
    # unnormlalize prediction
    pred = pred * target_std
    pred = pred + target_mean

plt.figure(figsize=(16,9))
plt.xlabel('Epoch --&gt;', size=20)
plt.ylabel('Loss --&gt;', size=20)
plt.xticks(size=20)
plt.yticks(size=20)
plt.title('Loss graph')
plt.plot(pred.numpy())
plt.plot(target.numpy())
plt.show()
</code></pre>
<p>If you comment the normalization out, you will see that the loss is huge (<code>~1e20</code>) and thus the gradients will also be very large. While this might be alright at the beginning of the training, I think you would have to play around with a learning rate scheduler in order to this the original dataset properly, so I would recommend to take a look at the normalization approach instead.</p>

Mona_Jalal: <p>Should I do the normalization in this cell? I am not sure exactly where I should perform this normalization. Also, is input here image and landmarks? or is landmarks targets?</p>
<pre><code class="lang-auto">network = Network()
network.cuda()    

criterion = nn.MSELoss()
optimizer = optim.Adam(network.parameters(), lr=0.0001)

loss_min = np.inf
num_epochs = 10

start_time = time.time()
for epoch in range(1,num_epochs+1):
    
    loss_train = 0
    loss_test = 0
    running_loss = 0
    
    
    network.train()
    print('size of train loader is: ', len(train_loader))

    for step in range(1,len(train_loader)+1):
        
        batch = next(iter(train_loader))
        images, landmarks = batch['image'], batch['landmarks']
        
        images = images.cuda()
        landmarks = landmarks.view(landmarks.size(0),-1).cuda() 
        
        predictions = network(images)
        
        # clear all the gradients before calculating them
        optimizer.zero_grad()
        
        # find the loss for the current step
        loss_train_step = criterion(predictions.float(), landmarks.float())
        
        print("type(loss_train_step) is: ", type(loss_train_step))
        
        print("loss_train_step.dtype is: ",loss_train_step.dtype)
                
        # calculate the gradients
        loss_train_step.backward()
        
        # update the parameters
        optimizer.step()
        
        loss_train += loss_train_step.item()
        running_loss = loss_train/step
        
        print_overwrite(step, len(train_loader), running_loss, 'train')
        
    network.eval() 
    with torch.no_grad():
        
        for step in range(1,len(test_loader)+1):
            
            batch = next(iter(train_loader))
            images, landmarks = batch['image'], batch['landmarks']        
            images = images.cuda()
            landmarks = landmarks.view(landmarks.size(0),-1).cuda()
        
            predictions = network(images)

            # find the loss for the current step
            loss_test_step = criterion(predictions, landmarks)

            loss_test += loss_test_step.item()
            running_loss = loss_test/step

            print_overwrite(step, len(test_loader), running_loss, 'Testing')
    
    loss_train /= len(train_loader)
    loss_test /= len(test_loader)
    
    print('\n--------------------------------------------------')
    print('Epoch: {}  Train Loss: {:.4f}  Test Loss: {:.4f}'.format(epoch, loss_train, loss_test))
    print('--------------------------------------------------')
    
    if loss_test &lt; loss_min:
        loss_min = loss_test
        torch.save(network.state_dict(), '../moth_landmarks.pth') 
        print("\nMinimum Test Loss of {:.4f} at epoch {}/{}".format(loss_min, epoch, num_epochs))
        print('Model Saved\n')
     
print('Training Complete')
print("Total Elapsed Time : {} s".format(time.time()-start_time))
</code></pre>
<p>I think landmarks is the target but I might be wrong. Any light in this direction is really appreciated.<br>
Also, doesn’t PyTorch have automatic method for normalization and whitening the data?</p>

ptrblck: <p><code>landmarks</code> in your code should be the targets.<br>
You could use <code>torchvision.transforms.Normalize</code> to whiten the data or just apply it manually as seen in my code snippet.</p>

Mona_Jalal: <p>I read somewhere in the notebook I have this:</p>
<pre><code class="lang-auto">data_transform = transforms.Compose([
        transforms.RandomSizedCrop(224),
        transforms.RandomHorizontalFlip(),
        transforms.ToTensor(),
        transforms.Normalize(mean=[0.485, 0.456, 0.406],
                             std=[0.229, 0.224, 0.225])
    ])
</code></pre>
<p>So, I am not sure what I set the mean and std to for either</p>
<pre><code class="lang-auto">
        batch = next(iter(train_loader))
        images, landmarks = batch['image'], batch['landmarks']
        
        images = images.cuda()
    
        landmarks = landmarks.view(landmarks.size(0),-1).cuda() 
        
        images = torchvision.transforms.Normalize(images, mean=[0.485, 0.456, 0.406], 
                                                          std=[0.229, 0.224, 0.225])
        landmarks = torchvision.transforms.Normalize(landmarks)
</code></pre>
<p>Also, should I normalize images and landmarks after doing a .cuda on them?</p>
<p>Additionally, how?</p>
<p>This is what I have from landmark:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/8/c8ebbfde8babc647190f61d69c39c1ba9675c7cb.png" data-download-href="https://discuss.pytorch.org/uploads/default/c8ebbfde8babc647190f61d69c39c1ba9675c7cb" title="Screenshot_2020-10-08_02-17-10"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/8/c8ebbfde8babc647190f61d69c39c1ba9675c7cb_2_690x283.png" alt="Screenshot_2020-10-08_02-17-10" data-base62-sha1="sFqAgTOejPGs3Hn1iiJCHsFICbp" width="690" height="283" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/8/c8ebbfde8babc647190f61d69c39c1ba9675c7cb_2_690x283.png, https://discuss.pytorch.org/uploads/default/optimized/3X/c/8/c8ebbfde8babc647190f61d69c39c1ba9675c7cb_2_1035x424.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/c/8/c8ebbfde8babc647190f61d69c39c1ba9675c7cb_2_1380x566.png 2x" data-dominant-color="EAEAEA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot_2020-10-08_02-17-10</span><span class="informations">1780×732 83 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<pre><code class="lang-auto">---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
&lt;ipython-input-21-88ae116843f0&gt; in &lt;module&gt;
     33 
     34         images = torchvision.transforms.Normalize(images, mean=[0.485, 0.456, 0.406], 
---&gt; 35                                                           std=[0.229, 0.224, 0.225])
     36         landmarks = torchvision.transforms.Normalize(landmarks)
     37 

TypeError: __init__() got multiple values for argument 'mean'
</code></pre>

ptrblck: <p>Depending on the shape of <code>landmarks</code> <code>Normalize</code> might not work.<br>
In your code you are assuming to normalize image tensors with 3 channels (thus the stats contain 3 values), which I don’t think is your use case.<br>
Calculate the <code>mean</code> and <code>std</code> (or <code>min</code> and <code>max</code>) from all training targets and normalize the <code>landmarks</code> tensors with them manually.</p>

Mona_Jalal: <p>so I am using this code block for transformed_dataset</p>
<pre><code class="lang-auto">transformed_dataset = MothLandmarksDataset(csv_file='moth_gt.csv',
                                           root_dir='.',
                                           transform=transforms.Compose([
                                               Rescale(256),
                                               RandomCrop(224),
                                               ToTensor()
                                           ]))

for i in range(len(transformed_dataset)):
    sample = transformed_dataset[i]

    print(i, sample['image'].size(), sample['landmarks'].size())

    if i == 3:
        break
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/f/8f15f4259329c7a470172d38e7bb23c68d0ce8d8.png" data-download-href="https://discuss.pytorch.org/uploads/default/8f15f4259329c7a470172d38e7bb23c68d0ce8d8" title="transformed_dataset"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/f/8f15f4259329c7a470172d38e7bb23c68d0ce8d8_2_690x272.png" alt="transformed_dataset" data-base62-sha1="kpNjssFFcxVZ8jo3r6LeCx3BdpK" width="690" height="272" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/f/8f15f4259329c7a470172d38e7bb23c68d0ce8d8_2_690x272.png, https://discuss.pytorch.org/uploads/default/optimized/3X/8/f/8f15f4259329c7a470172d38e7bb23c68d0ce8d8_2_1035x408.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/8/f/8f15f4259329c7a470172d38e7bb23c68d0ce8d8_2_1380x544.png 2x" data-dominant-color="F1F2F2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">transformed_dataset</span><span class="informations">2222×878 95.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Does PyTorch have a utility function for doing data normalization? Is there a simple tutorial you could please link me to so I could learn? I am not sure how to normalize my data using <code>transform</code></p>

ptrblck: <p>It depends what and how you would like to normalize the data.</p>
<p>E.g. image tensors can be normalized using <code>transforms.Normalize</code>, which will calculate their z-score using the passed <code>mean</code> and <code>std</code>. This would change the values of the image tensor, such that they have a zero mean and unit variance (if the right stats are passed).</p>
<p>However, the landmarks would represent a pixel position inside the image, if I’m not mistaken.<br>
I.e. how would you like to normalize them?<br>
You could divide the values by their max, such that all landmark values are in the range <code>[0, 1]</code>, but this would of course mean that you have to “denormalize” them if you want to plot them into the image and calculate the “real” loss.<br>
Nevertheless, normalizing the target (and denormalizing afterwards) might help as shown in the linked thread.</p>

Mona_Jalal: <p>should I normalize across all the examples in training set or all examples in my entire dataset? I meant should I find the mean and std of images across all the training set images or all the images in the datasets? Is there an example that shows how to do this from scratch not how to use some mean and std already known? The transfer learning tutorial for example uses an already known mean and std for normalization of images.<br>
<code>transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])</code></p>
<aside class="onebox allowlistedgeneric">
  <header class="source">
      <a href="https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html" target="_blank" rel="noopener nofollow ugc">pytorch.org</a>
  </header>
  <article class="onebox-body">
    <img src="" class="thumbnail" width="" height="">

<h3><a href="https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html" target="_blank" rel="noopener nofollow ugc">Transfer Learning for Computer Vision Tutorial — PyTorch Tutorials 1.6.0...</a></h3>



  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


Mona_Jalal: <p>also, tried this and got error</p>
<pre><code class="lang-auto">transformed_dataset = MothLandmarksDataset(csv_file='moth_gt.csv',
                                           root_dir='.',
                                           transform=transforms.Compose([
                                               Rescale(256),
                                               RandomCrop(224),
                                               transforms.Normalize(mean = [ 0.485, 0.456, 0.406 ],
                                                        std = [ 0.229, 0.224, 0.225 ]),
                                               ToTensor()
                                           ]))

for i in range(len(transformed_dataset)):
    sample = transformed_dataset[i]

    print(i, sample['image'].size(), sample['landmarks'].size())

    if i == 3:
        break
</code></pre>
<p>error is:</p>
<pre><code class="lang-auto">---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
&lt;ipython-input-81-eb8dc46e0284&gt; in &lt;module&gt;
     10 
     11 for i in range(len(transformed_dataset)):
---&gt; 12     sample = transformed_dataset[i]
     13 
     14     print(i, sample['image'].size(), sample['landmarks'].size())

&lt;ipython-input-48-9d04158922fb&gt; in __getitem__(self, idx)
     30 
     31         if self.transform:
---&gt; 32             sample = self.transform(sample)
     33 
     34         return sample

~/anaconda3/lib/python3.7/site-packages/torchvision/transforms/transforms.py in __call__(self, img)
     59     def __call__(self, img):
     60         for t in self.transforms:
---&gt; 61             img = t(img)
     62         return img
     63 

~/anaconda3/lib/python3.7/site-packages/torchvision/transforms/transforms.py in __call__(self, tensor)
    210             Tensor: Normalized Tensor image.
    211         """
--&gt; 212         return F.normalize(tensor, self.mean, self.std, self.inplace)
    213 
    214     def __repr__(self):

~/anaconda3/lib/python3.7/site-packages/torchvision/transforms/functional.py in normalize(tensor, mean, std, inplace)
    278     """
    279     if not torch.is_tensor(tensor):
--&gt; 280         raise TypeError('tensor should be a torch tensor. Got {}.'.format(type(tensor)))
    281 
    282     if tensor.ndimension() != 3:

TypeError: tensor should be a torch tensor. Got &lt;class 'dict'&gt;.
</code></pre>

ptrblck: <p>You should use the training dataset statistics only.<br>
To get the stats for this dataset, you could load the complete dataset in a single tensor (if it fits into your memory) and calculate the stats or iterate the dataset and calculate the mean and std from the running stats.</p>
<p>The transform error is raised, since a tensor is expected but you’ve passed a <code>dict</code> so I guess the image tensor might be wrapped inside the <code>dict</code> and you would have to access it.</p>

FA_mn: <aside class="quote no-group" data-username="ptrblck" data-post="5" data-topic="96648">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<pre><code class="lang-auto">pred = model(input)
    # unnormlalize prediction
    pred = pred * target_std
    pred = pred + target_mean
</code></pre>
</blockquote>
</aside>
<p>Hello,<br>
Thanks for the useful information and suggestion.<br>
I have defined a custom loss function that depends on the first and second derivatives of the output of the NN with respect to the input.<br>
(like PDE in <a href="https://discuss.pytorch.org/t/first-and-second-derivates-of-the-output-with-respect-to-the-input-inside-a-loss-function/99757">https://discuss.pytorch.org/t/first-and-second-derivates-of-the-output-with-respect-to-the-input-inside-a-loss-function/99757</a>)</p>
<p>If I normalize the target and train the model, how can I unnormalize the predictions?<br>
Because of the first and second derivatives of the output of the NN w. r. t. the input used in loss function, I think</p>
<pre><code class="lang-auto">pred = model(input) 
pred = pred * target_std 
pred = pred + target_mean
</code></pre>
<p>does not work correctly.<br>
Could you please inform me about this question?</p>