Passing nn output to custom loss function doesn't train

username: srslysirius
cooked: <p>Sorry to post here, I’m new to this and tried searching all day and can’t quite figure out what to do.</p>
<p>playgame() is a 600-line function that takes a bunch of arguments (valued between 0-1) as “moves” and outputs a score. I have excluded it here, but can pastebin it or something if needed.</p>
<pre><code>class TwoLayerNet(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc1 = nn.Linear(438,100)
        self.fc2 = nn.Linear(100,438)
    def forward(self, x):
        x = F.sigmoid(self.fc1(x))
        x = F.sigmoid(self.fc2(x))
        return x

m = TwoLayerNet()
loss_fn = nn.MSELoss()
optimizer = optim.Adam(m.parameters(), lr=0.01)

training_epochs = 25
fitnessgoal = torch.tensor(1200, dtype=torch.float)
blanktensor = torch.zeros([1, 438])


def myloss(output, fitnessgoal):
    orders = output.tolist()[0]
    fitness = torch.tensor(playgame(*orders), dtype=torch.float)
    loss = loss_fn(fitness, fitnessgoal)  
    loss.requires_grad = True
    loss.retain_grad()
    return loss

for i in range(training_epochs):

    # forward pass
    fpass = m(blanktensor)

    # compute and print loss
    loss = myloss(fpass, fitnessgoal)
    loss.requires_grad = True
    print(i,fitness)
    print(loss.grad)

    # reset gradients
    optimizer.zero_grad()

    # backwards pass
    loss.backward()

    # step the optimizer - update the weights
    optimizer.step()
</code></pre>
<p>Doing it this way results in the loss remaining exactly the same each epoch. It seems that loss.backward() and optimizer.step() aren’t actually doing anything. loss.grad results in None, which seems to confirm this.</p>
<p>As far as I can gather this is because it can’t back propagate through the playgame() function. I imagine I’m probably severing things when I convert the output from a tensor to a regular list. But playgame() can’t use a tensor as argument, and even if it could, would that even work as a loss function?</p>
<p>I’m trying to train the network to refine a 438 length list of floats (between 0 and 1) that result in the highest possible score when passed through playgame(). Is it possible to do stuff like this with Pytorch?</p>

username: ptrblck
cooked: <p>If you are using just PyTorch functions, Autograd will be able to create the backward pass automatically. However, if you are using plain Python or bumpy methods, you might break the computation graph, which will stop the backward pass.</p>
<p>Which methods are you using, which are not implemented in PyTorch?</p>

username: srslysirius
cooked: <p>Methods? I believe the only kind used in playgame() are .append() for various lists. If you care to look, I’ve pasted the function here: <a href="https://pastebin.com/W5anDqcu" rel="nofollow noopener">https://pastebin.com/W5anDqcu</a></p>
<p>Aside from that, you can see that I’m using .tolist() inside the custom loss function</p>

username: ptrblck
cooked: <p>I’m not sure, what the method is doing, but at some point you would have to calculate the loss.<br>
Could you check, which operations are included in the loss calculations?</p>

username: srslysirius
cooked: <p>Sure. After the forward pass, the network outputs a tensor of [1, 481] size. Then this output is used in my custom loss function:</p>
<pre><code>def myloss(output, fitnessgoal):
    orders = output.tolist()[0]
    fitness = torch.tensor(playgame(*orders), dtype=torch.float)
    loss = loss_fn(fitness, fitnessgoal)  
    loss.requires_grad = True
    loss.retain_grad()
    return loss</code></pre>

username: srslysirius
cooked: <p>I removed the myloss() function altogether and fixed the playgame() function so that it can accept a tensor as an argument. The training loop now looks like this:</p>
<pre><code>m = TwoLayerNet()
loss_fn = nn.MSELoss()
optimizer = optim.Adam(m.parameters(), lr=0.01)

training_epochs = 10
fitnessgoal = torch.tensor(1200, dtype=torch.float)
blanktensor = torch.zeros([1, 438])

   
for i in range(training_epochs):

    fpass = m(blanktensor)

    loss = loss_fn(playgame(fpass), fitnessgoal)
    loss.requires_grad = True
    print(i,fitness,loss)
    print(loss.grad)

    optimizer.zero_grad()

    loss.backward()

    optimizer.step()
</code></pre>
<p>Unfortunately, the result is pretty much the same. 10 epochs:</p>
<pre><code>0 tensor(744.) tensor(207936., requires_grad=True)
None
1 tensor(744.) tensor(207936., requires_grad=True)
None
2 tensor(744.) tensor(207936., requires_grad=True)
None
3 tensor(744.) tensor(207936., requires_grad=True)
None
4 tensor(744.) tensor(207936., requires_grad=True)
None
5 tensor(744.) tensor(207936., requires_grad=True)
None
6 tensor(744.) tensor(207936., requires_grad=True)
None
7 tensor(744.) tensor(207936., requires_grad=True)
None
8 tensor(744.) tensor(207936., requires_grad=True)
None
9 tensor(744.) tensor(207936., requires_grad=True)
None
</code></pre>
<p>The loss never changes, and gradient is None. I thought that perhaps the playgame() function was still to blame for this, so I added this to the end of it:</p>
<pre><code>fitness = torch.tensor(land, dtype=torch.float)
fitness.requires_grad = True

return(fitness)
</code></pre>
<p>That resulted in the error “you can only change requires_grad flags of leaf variables”. I don’t exactly know what this means, but it seems that running a tensor through the playgame() function is disconnecting things somehow. Maybe Pytorch wasn’t meant to be used like this.</p>

username: ptrblck
cooked: <p>It still seems you are detaching the computation graph somewhere.<br>
If you want to backpropagate through the model, your forward and loss function should use tensors and contain PyTorch methods. Otherwise you would have to implement the <code>backward</code> function manually.</p>
<p>In your updated <code>myloss</code> function, you are re-wrapping the output of <code>playgame</code> in a tensor, which will also break the graph.</p>

username: srslysirius
cooked: <p>I’m not sure what you mean about the updated myloss(), I removed that altogether. Are you referring to this?</p>
<pre><code>fitness = torch.tensor(land, dtype=torch.float)
fitness.requires_grad = True

return(fitness)
</code></pre>
<p>That’s from the playgame() function. I did this because otherwise I got an error about integers not having a ‘size’ attribute.</p>
<p>So far I’ve felt pretty sure that it’s the playgame() function detaching things somehow, but I also just tested the forward pass itself. It looks like a pretty standard network architecture, and it’s applying weights to the zerod tensor. But when I print the .grad attribute for its output, that also gets None.</p>
<pre><code>class TwoLayerNet(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc1 = nn.Linear(438,100)
        self.fc2 = nn.Linear(100,438)
    def forward(self, x):
        x = F.sigmoid(self.fc1(x))
        x = F.sigmoid(self.fc2(x))
        return x

m = TwoLayerNet()
for i in range(training_epochs):

    fpass = m(blanktensor)
    print(fpass.grad)
</code></pre>
<p>So maybe there’s some kind of issue with this as well?</p>