Why my predictions are all zeros during the training phase of my Neural Network?

username: Sunakutra76
cooked: <p>Hello,</p>
<p>I am a beginner with PyTorch, and I’m working on a small project which consists in predicting some output. To do this I’m using a Neural Network that has a few layers with simple transforms.</p>
<p>When I try to run my code I’m faced with the following problem : every prediction of my neural network is 0 (sometimes it gives other values in the first batch when I’m lucky enough…)</p>
<p>Here is the code of my project :</p>
<pre><code class="lang-auto">import torch
from torch import nn
from torch.utils.data import DataLoader
from torch.utils.data import Dataset
from torchvision.transforms import Lambda, Compose

import matplotlib.pyplot as plt
import numpy as np

n = 20      #Nombre de variables de direction résumant la variance du payoff
d = 20      #Dimension de l'option

N_data_training = 5000      #Taille de la base de donnée utile à l'apprentissage du réseau de neurones
N_data_test = 2500      #Taille de la base de donnée utile au test de la précision du réseau de neurones entrainé

type_product = "basket call"    #Type de produit dérivé
S0 = 100      #Spot price
K = 100     #Strike
T = 1     #Maturité
r = 0.05      #Taux d'intérêt sans risque
sigma = 0.2     #Volatilité
# Nmc = 1000      #Nombre de trajectoires générées pour l'estimation de Monte-Carlo
learning_rate = 1e-3

device = "cuda" if torch.cuda.is_available() else "cpu"
print("Using {} device".format(device))

def payoff_derivative(N_data, type_product, d, S0, r, sigma, T): #### pay-offs test
  payoff = np.zeros(N_data)
  Z = np.zeros((N_data, d))

  if type_product == "basket call":
    for i in range(N_data):
      Z[i, :] = S0*np.exp((r-0.5*sigma**2)*T + sigma*np.sqrt(T)*np.random.randn(d))
      payoff[i] = np.maximum((1/d)*np.sum(Z[i, :], axis=0) - K, 0)

  return (Z, payoff)

pure_data_training = payoff_derivative(N_data_training, type_product, d, S0, r, sigma, T) ##### pay-offs pour l'apprentissage du réseau de neurones
pure_data_test = payoff_derivative(N_data_test, type_product, d, S0, r, sigma, T) ##### pay-offs pour le test de précision


class Creation_Dataset(Dataset):
  def __init__(self, pure_data):
    self.Z = pure_data[0]
    self.payoff = pure_data[1]

  def __len__(self):
    return len(self.payoff)

  def __getitem__(self, idx):
    return torch.tensor(self.Z[idx],dtype=torch.float), torch.tensor(self.payoff[idx],dtype=torch.float)


dataset_training = Creation_Dataset(pure_data_training)
dataset_test = Creation_Dataset(pure_data_test)

# Creation des data loaders
batch_size = 100
train_dataloader = DataLoader(dataset_training, batch_size)
test_dataloader = DataLoader(dataset_test, batch_size)

for X, y in train_dataloader:
    print("Shape of X : ", X.shape, X.dtype)
    print("Shape of y: ", y.shape, y.dtype)
    print(X[0])
    print(y[0])
    break

class NeuralNetwork(nn.Module):
    def __init__(self): ######méthode dans laquelle on définit les couches du réseau
        super(NeuralNetwork, self).__init__()
        self.flatten = nn.Flatten() ########## convertit en tenseur 1D
        self.linear_relu_stack = nn.Sequential(
            nn.Linear(d, n, bias=False),
            nn.Linear(n, 6),
            nn.ReLU(),
            nn.Linear(6, 15),
            nn.ReLU(),
            nn.Linear(15, 10),
            nn.ReLU(),
            nn.Linear(10, 5),
            nn.ReLU(),
            nn.Linear(5, 1),
            nn.ReLU()
          )

    def forward(self, x):
        logits = self.linear_relu_stack(x)
        return torch.reshape(logits, (-1,))

model = NeuralNetwork().to(device)
print(model)

loss_fn = nn.MSELoss(reduction='sum')
optimizer = torch.optim.SGD(model.parameters(), lr=1e-3)

def train_loop(dataloader, model, loss_fn, optimizer):
    size = len(dataloader.dataset)
    for batch, (X, y) in enumerate(dataloader):
        # Compute prediction and loss
        print("Shape of X : ", X.shape, X.dtype)
        print("Shape of y: ", y.shape, y.dtype)
        pred = model(X)
        print(pred)
        loss = loss_fn(pred, y)

        # Backpropagation
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        if batch % 100 == 0:
            loss, current = loss.item(), batch * len(X)
            print(f"loss: {loss:&gt;7f}  [{current:&gt;5d}/{size:&gt;5d}]")

def test(dataloader, model):
    model.eval()
    test_loss, correct = 0, 0
    with torch.no_grad():
        for X, y in dataloader:
            X, y = X.to(device), y.to(device)
            pred = model(X)
            test_loss += loss_fn(pred, y).item()
            correct += float((pred == y).sum()) 
    size = len(dataloader.dataset)
    test_loss /= size
    correct /= size
    print(f"Accuracy: {100*correct:.2f}%, Avg loss (sqrt(MSE)) : {np.sqrt(test_loss):.3f} \n")

epochs = 2
for t in range(epochs):
    print(f"Epoch {t+1}\n-------------------------------")
    train(train_dataloader, model, loss_fn, optimizer)
    test(test_dataloader, model)
</code></pre>
<p>So, using a DataLoader that creates batches of batch_size, I train my Neural Network in the train_loop() function and I print out my predictions’ tensor, and it turns out that all my predictions are zeros. I’ve seen a few topics with people facing this kind of problems with classification problems, but I’m still trying to solve it for predicting correct outputs.</p>
<p>Thank you for your time</p>

username: KFrank
cooked: <p>Hi Sunakutra!</p>
<p>I haven’t looked at all of your code, but I do see some issue, noted<br>
in line, below:</p>
<aside class="quote no-group quote-modified" data-username="Sunakutra76" data-post="1" data-topic="119991" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/b782af/48.png" class="avatar"> Sunakutra76:</div>
<blockquote>
<pre><code class="lang-auto">class NeuralNetwork(nn.Module):
        self.linear_relu_stack = nn.Sequential(
            nn.Linear(d, n, bias=False),
            nn.Linear(n, 6),
</code></pre>
</blockquote>
</aside>
<p>This won’t “break” your model (or cause it to return zeroes), but<br>
two <code>Linear</code>s in a row – without an intervening non-linearity – are<br>
redundant, and should be replaced by a single <code>Linear (d, 6)</code>.</p>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto">            ...
            nn.Linear(5, 1),
            nn.ReLU()
          )
</code></pre>
</blockquote>
</aside>
<p>This could be your problem.  Using a final <code>ReLU</code> layer to output your<br>
predictions is a poor choice.  If your last <code>Linear</code> happens to generate<br>
negative values, <code>ReLU</code> will output zero and will have zero gradient<br>
when you backpropagate, so your optimizer won’t know in what<br>
direction to move to get you “unstuck” out of the flat (zero gradient)<br>
part of <code>ReLU</code>.</p>
<p>At the least you could try something like <a href="https://pytorch.org/docs/stable/generated/torch.nn.LeakyReLU.html#torch.nn.LeakyReLU" rel="noopener nofollow ugc">LeakyReLU</a>, but I would<br>
recommend removing the final <code>ReLU</code>, and letting your <code>MSELoss</code> and<br>
backpropagation “learn” to predict positive values.  (Here I assume<br>
that you a training with “target” prices that are always positive.)</p>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto">            pred = model(X)
            test_loss += loss_fn(pred, y).item()
            correct += float((pred == y).sum()) 
</code></pre>
</blockquote>
</aside>
<p>Your accuracy prediction will fail here.  You are testing for <em>exact</em><br>
equality between two floating-point number which will almost never<br>
be true (except for some special cases).  Since you’re predicting<br>
continuous values (rather than classes or categories) you should<br>
use something like your <code>MSELoss</code> as a figure of merit.  You could<br>
also count the number of “correct” predictions by defining “correct”<br>
to be a prediction that is within a certain absolute amount or within<br>
a certain percentage of the ground-truth value.</p>
<p>Best.</p>
<p>K. Frank</p>

username: Sunakutra76
cooked: <p>Hi KFrank,</p>
<p>Thank you four your answer, I fixed all the points you have mentioned, and I removed the last nn.ReLU()  in my model, and tried to put alternatively Tanh and ReLU for the remaining activation functions. It seems that in fact the model returns the same prediction for every input of my batches during the training phase, as shown in the following test with batches of batch_size = 10 :</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/2/92280bd30219c5c7e9f7075473153be97dfcee7e.png" data-download-href="https://discuss.pytorch.org/uploads/default/92280bd30219c5c7e9f7075473153be97dfcee7e" title="pb"><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/2/92280bd30219c5c7e9f7075473153be97dfcee7e.png" alt="pb" data-base62-sha1="kQXvzydC5BgH5kzqFXFNQrQcFMW" width="575" height="500" data-dominant-color="F6F6F6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">pb</span><span class="informations">628×546 16.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>At least I have positive values, but I still can’t see what is the problem here, I even tried to print out the weights of the model for each linear transform, or avoid using ReLU but the problem still holds.</p>
<p>Thank you</p>

username: cskarthik7
cooked: <p>Also I would like to mention that I tried to run your code and your labels are floating point values whereas the labels should be some fixed values where the model is able to create a mapping function between the input and the output. <a class="mention" href="/u/sunakutra76">@Sunakutra76</a></p>

username: KFrank
cooked: <p>Hi Sunakutra!</p>
<aside class="quote no-group" data-username="Sunakutra76" data-post="3" data-topic="119991" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/b782af/48.png" class="avatar"> Sunakutra76:</div>
<blockquote>
<p>It seems that in fact the model returns the same prediction for every input of my batches during the training phase, as shown in the following test with batches of batch_size = 10 :</p>
</blockquote>
</aside>
<p>(As an aside, please don’t post screenshots of textual information.  It<br>
breaks accessibility, searchability, and copy-paste.)</p>
<p>It works for me, at least in so far as I don’t get the same prediction for<br>
every input.</p>
<p>I copied your code, made a couple of the changes you made, and I can<br>
run it and get non-identical predictions.</p>
<p>Two suggestions:  You might probe the <code>bias</code> in your final <code>Linear</code><br>
layer.  If the output of the preceding layer gets “stuck” at zero, then<br>
all of your final-layer predictions will just be the final layer’s `bias’.</p>
<p>Also, verify that your batch doesn’t consist (for some weird reason)<br>
of 10 copies of the same input.</p>
<p>Here’s a script that shows non-identical predictions (for random input<br>
data) and shows how to look at the model layers and the final bias:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
from torch import nn
print (torch.__version__)

_ = torch.manual_seed (2021)

n = 20
d = 20
batch_size = 10

class NeuralNetwork(nn.Module):
    def __init__(self): ######methode dans laquelle on definit les couches du reseau
        super(NeuralNetwork, self).__init__()
        self.flatten = nn.Flatten() ########## convertit en tenseur 1D
        self.linear_relu_stack = nn.Sequential(
            # nn.Linear(d, n, bias=False),
            # nn.Linear(n, 6),
            nn.Linear(d, 6),
            nn.ReLU(),
            nn.Linear(6, 15),
            nn.ReLU(),
            nn.Linear(15, 10),
            nn.ReLU(),
            nn.Linear(10, 5),
            nn.ReLU(),
            nn.Linear(5, 1),
            # nn.ReLU()
        )
    
    def forward(self, x):
        logits = self.linear_relu_stack(x)
        return torch.reshape(logits, (-1,))

model = NeuralNetwork()

batch_of_samples = torch.randn (batch_size, d)

print ("model (batch_of_samples) = ...")
print (model (batch_of_samples))

print ("model = ...")
print (model)
print ("model._modules['linear_relu_stack'][8] =", model._modules['linear_relu_stack'][8])
print ("model._modules['linear_relu_stack'][8].bias =", model._modules['linear_relu_stack'][8].bias)
</code></pre>
<p>And here is its output:</p>
<pre><code class="lang-plaintext">1.7.1
model (batch_of_samples) = ...
tensor([0.3791, 0.4269, 0.3727, 0.4019, 0.3934, 0.4013, 0.3771, 0.3855, 0.3729,
        0.3700], grad_fn=&lt;ViewBackward&gt;)
model = ...
NeuralNetwork(
  (flatten): Flatten(start_dim=1, end_dim=-1)
  (linear_relu_stack): Sequential(
    (0): Linear(in_features=20, out_features=6, bias=True)
    (1): ReLU()
    (2): Linear(in_features=6, out_features=15, bias=True)
    (3): ReLU()
    (4): Linear(in_features=15, out_features=10, bias=True)
    (5): ReLU()
    (6): Linear(in_features=10, out_features=5, bias=True)
    (7): ReLU()
    (8): Linear(in_features=5, out_features=1, bias=True)
  )
)
model._modules['linear_relu_stack'][8] = Linear(in_features=5, out_features=1, bias=True)
model._modules['linear_relu_stack'][8].bias = Parameter containing:
tensor([0.2732], requires_grad=True)
</code></pre>
<p>If the problem persists, could you post a complete, runnable script<br>
that sets <code>torch.manual_seed</code> (so we can reproduce the initialization<br>
of your model weights), builds an input tensor from explicit values (so<br>
that when we copy-paste your script we’ll be building the same tensor),<br>
and prints out one batch of identical predictions?</p>
<p>Best.</p>
<p>K. Frank</p>

username: KFrank
cooked: <p>Hi Karthik!</p>
<aside class="quote no-group" data-username="cskarthik7" data-post="4" data-topic="119991" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/cskarthik7/48/23944_2.png" class="avatar"> cskarthik7:</div>
<blockquote>
<p>your labels are floating point values whereas the labels should be some fixed values</p>
</blockquote>
</aside>
<p>I don’t think this is correct.</p>
<p>It appears that Sunakutra is not building a classifier (where the labels<br>
would be fixed, integer class labels), but, rather, is performing a<br>
“regression,” where both the predictions and <code>target</code> (not really labels)<br>
are continuous variables that are supposed to be close to one another.</p>
<p>He is using <code>MSELoss</code>, which is appropriate for training continuous<br>
predictions against continuous <code>target</code>s (rather than something like<br>
<code>CrossEntropyLoss</code> which would be appropriate for building a classifier<br>
that has fixed categorical labels for the <code>target</code>).</p>
<p>Best.</p>
<p>K. Frank</p>