Manual loss accumulation

William_Caster: <p>Hi there, I’m trying to accumulate the loss manually like this:</p>
<pre><code class="lang-auto"># -- training network ----------------------------------------

num_epochs = 5

loss_function = nn.MSELoss()

optimizer = optim.SGD(model.parameters(), lr = 0.01)

model.train()

for epoch in range(num_epochs):

    _epoch_loss = []

    for i in range(len(training_data['X'][0:20])):            # all cartpole simulations (each sim. is an episode).

        optimizer.zero_grad()

        accumulated_MSE = 0.0

        model[0].hebb = torch.zeros((30, 4), requires_grad = False)
        model[2].hebb = torch.zeros((30, 30), requires_grad = False)
        model[4].hebb = torch.zeros((1, 30), requires_grad = False)

        for j in range(len(training_data['X'][i])):            # all time steps of simulation i.

            _input = torch.tensor(np.array([training_data['X'][i][j]]), dtype = torch.float32)
            _target = torch.tensor(np.array([training_data['Y'][i][j]]), dtype = torch.float32)

            prediction = model(_input)

            prediction = torch.tensor(np.array(prediction.tolist()[0]), dtype = torch.float32)

            accumulated_MSE += loss_function(prediction, _target).item()

        accumulated_MSE_tensor = torch.tensor(accumulated_MSE, dtype=torch.float32, requires_grad=True)

        accumulated_MSE_tensor.backward()
        optimizer.step()

        model[0].hebb.detach_()
        model[2].hebb.detach_()
        model[4].hebb.detach_()

        # episode ends.

        _epoch_loss.append(accumulated_MSE)

    print(f'epoch {epoch} loss: {np.mean(_epoch_loss)}')
</code></pre>
<p>The problem is that the loss is not changing between epochs. What’s the correct way to accumulate the loss manually?</p>

AlphaBetaGamma96: <aside class="quote no-group" data-username="William_Caster" data-post="1" data-topic="183985">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/william_caster/48/61958_2.png" class="avatar"> William_Caster:</div>
<blockquote>
<pre><code class="lang-auto">        accumulated_MSE_tensor = torch.tensor(accumulated_MSE, dtype=torch.float32, requires_grad=True)
</code></pre>
</blockquote>
</aside>
<p>You’re breaking your graph here. Don’t re-wrap tensors. Also, you shouldn’t be using the <code>requires_grad=True</code> on new tensors, as that destroys the history of operations leading to no gradients</p>

William_Caster: <p>If I simply do <strong>accumulated_MSE.backward()</strong> I get <strong>AttributeError: ‘float’ object has no attribute 'backward</strong>. How should I do it?</p>

AlphaBetaGamma96: <aside class="quote no-group" data-username="William_Caster" data-post="1" data-topic="183985">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/william_caster/48/61958_2.png" class="avatar"> William_Caster:</div>
<blockquote>
<pre><code class="lang-auto">            accumulated_MSE += loss_function(prediction, _target).item()
</code></pre>
</blockquote>
</aside>
<p>remove <code>.item()</code>, its casting a Tensor to float</p>

William_Caster: <p>This does not work. If I remove the .item() and don’t cast it to a new tensor like you suggested (code bellow) I get the error:</p>
<p><strong>Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass<br>
RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn</strong></p>
<pre><code class="lang-auto">for epoch in range(num_epochs):

    _epoch_loss = []

    for i in range(len(training_data['X'][0:20])):            # all cartpole simulations (each sim. is an episode).

        optimizer.zero_grad()

        accumulated_MSE = 0.0

        model[0].hebb = torch.zeros((30, 4), requires_grad = False)
        model[2].hebb = torch.zeros((30, 30), requires_grad = False)
        model[4].hebb = torch.zeros((1, 30), requires_grad = False)

        for j in range(len(training_data['X'][i])):            # all time steps of simulation i.

            _input = torch.tensor(np.array([training_data['X'][i][j]]), dtype = torch.float32)
            _target = torch.tensor(np.array([training_data['Y'][i][j]]), dtype = torch.float32)

            prediction = model(_input)

            prediction = torch.tensor(np.array(prediction.tolist()[0]), dtype = torch.float32)

            accumulated_MSE += loss_function(prediction, _target)

        accumulated_MSE.backward()
        optimizer.step()
</code></pre>

AlphaBetaGamma96: <p>Do you have a <code>grad_fn</code> for <code>accumlated_MSE</code>? Can you see if your <code>model.parameters()</code> have a <code>grad_fn</code>?</p>

William_Caster: <p>accumlated_MSE has no grad_fn, that’s why I was wrapping it around a tensor with require_grad = True. The model parameters do have a grad_fn.</p>

AlphaBetaGamma96: <aside class="quote no-group" data-username="William_Caster" data-post="5" data-topic="183985">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/william_caster/48/61958_2.png" class="avatar"> William_Caster:</div>
<blockquote>
<pre><code class="lang-auto">            prediction = torch.tensor(np.array(prediction.tolist()[0]), dtype = torch.float32)
</code></pre>
</blockquote>
</aside>
<p>re-wrapping here breaks the gradient flow you should pass the output from <code>model</code> to <code>accumulated_MSE</code></p>

William_Caster: <p>Thx a lot <a class="mention" href="/u/alphabetagamma96">@AlphaBetaGamma96</a> ! That was indeed the problem. It’s solved now <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>