Loss remains the same during training

dimpo: <p>I’m facing a weird problem using the latest PyTorch version (<code>1.12.1</code>) in a JupyterLab environment.</p>
<p>I have this model definition:</p>
<pre><code class="lang-auto">class MyModel(nn.Module):
    def __init__(self):
        super().__init__()
        
        self.fc_1 = nn.Linear(1, 100)
        self.fc_2 = nn.Linear(100, 1)
        self.relu = nn.ReLU()
    
    def forward(self, x):
        x = self.fc_1(x)
        x = self.relu(x)
        x = self.fc_2(x)
        return self.relu(x)

net = MyModel()
</code></pre>
<p>The objective:</p>
<pre><code class="lang-auto">mse = torch.nn.MSELoss()
</code></pre>
<p>And the optimizer:</p>
<pre><code class="lang-auto">optimizer = torch.optim.SGD(net.parameters(), lr=1e-2)
</code></pre>
<p>I’m training the model using the following simple training loop:</p>
<pre><code class="lang-auto">def train(epochs, x, y):
    for epoch in range(epochs):
        # forward pass
        pred = net(x)
        # compute the loss
        loss = mse(pred, y)
        # backward pass
        loss.backward()
        # optimization step
        optimizer.step()
        # zero out the gradients to avoid gradient accumulation
        optimizer.zero_grad()

        print(f"Epoch: {epoch}\t Loss: {loss}")

train(5)
</code></pre>
<p>The really weird issue is that the first time I run the process, the model is not learning anything, and the loss remains the same. However, if I run the cell with the model definition and instantiation again, the loss function and the optimizer everything works.</p>
<p>Any clues?</p>

KFrank: <p>Hi Dimitris!</p>
<aside class="quote no-group" data-username="dimpo" data-post="1" data-topic="163256" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/dimpo/48/53921_2.png" class="avatar"> dimpo:</div>
<blockquote>
<p>I’m facing a weird problem using the latest PyTorch version (<code>1.12.1</code>)</p>
</blockquote>
</aside>
<p>Could you post a short, fully-self-contained, runnable script that reproduces<br>
your issue together with the output you get?</p>
<p>(It seems like you’ve already posted most of what would be in such a script.)</p>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto">def train(epochs):
    for epoch in range(epochs, x, y):
        # forward pass
        pred = net(x)
        # compute the loss
        loss = mse(pred, y)
</code></pre>
</blockquote>
</aside>
<p>This looks odd here.  What are <code>x</code> and <code>y</code>?  In <code>range (epochs, y, y)</code>, <code>x</code> and<br>
<code>y</code> would typically be integers, while in <code>net (x)</code> and <code>mse (pred, y)</code>, <code>x</code> and<br>
<code>y</code> would typically be pytorch tensors.  I don’t see how this code could run.</p>
<p>Best.</p>
<p>K. Frank</p>

dimpo: <p><a class="mention" href="/u/kfrank">@KFrank</a>, you’re right. This was a copy-paste error. I edited the example. <code>x</code> and <code>y</code> are two PyTorch tensors and they are now moved to the <code>train</code> function’s signature. I obtain them like this:</p>
<pre><code class="lang-auto">x = torch.linspace(-2, 2, steps=20)[:, None]
x = x.float()
y = add_noise(f(x), .3, 1.5)
y = y.float()
</code></pre>
<p>where <code>f</code> is</p>
<pre><code class="lang-auto">def f(x):
    return 3*x**2 + 2*x + 1
</code></pre>
<p>and <code>add_noise</code> is</p>
<pre><code class="lang-auto">def noise(x, scale):
    return np.random.normal(scale=scale, size=x.shape)

def add_noise(x, mult, add):
    return x * (1 + noise(x, mult)) + noise(x, add)
</code></pre>

ptrblck: <aside class="quote no-group" data-username="dimpo" data-post="1" data-topic="163256">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/dimpo/48/53921_2.png" class="avatar"> dimpo:</div>
<blockquote>
<p>The really weird issue is that the first time I run the process, the model is not learning anything, and the loss remains the same. However, if I run the cell with the model definition and instantiation again, the loss function and the optimizer everything works.</p>
</blockquote>
</aside>
<p>This sounds like a flaky training where the convergence fails randomly. Did you try rerunning the code using different seeds to check the convergence rate?</p>
<p>Also, based on your post it seems the output could be negative if the noise is large enough while the last <code>relu</code> usage would clip the output so you might want to remove it.</p>