Propagating loss on multiple output layers

username: shavitamit
cooked: <p>I’m following the tutorial here: <a href="http://pytorch.org/tutorials/intermediate/char_rnn_classification_tutorial.html" rel="nofollow noopener">http://pytorch.org/tutorials/intermediate/char_rnn_classification_tutorial.html</a></p>
<p>In the image an input (and hidden layer) are combined and go through the network where both the current state (hidden layer) and output layer are generated.</p>
<p>My question is: what if I want to add another route to the network (i.e., predict some binary label <em>as well as</em> continue to do the RNN that’s in the image).  The network would essentially predict two things, (1) a binary label, and (2) the next word in the sequence.</p>
<p>Perhaps a simplified version of my question in code would be simpler to understand.  Given the following network architecture:</p>
<pre><code>class Model(nn.Module):

    def __init__(self, inputs):
        
        super(Model, self).__init__()
        
        self.linear1 = nn.Linear(inputs,100)
        
        self.l2o1 = nn.Linear(100,2)
        self.l2o2 = nn.Linear(100,2)
        
    def forward(self, inputs):
        
        o = self.linear1(inputs)
        
        o1 = F.log softmax(self.l2o1(o))
        o2 = F.log_softmax(self.l2o2(o))
        
        return o1, o2
</code></pre>
<p>How can I properly propagate the error on this network that generates two binary predictions?  Is this possible?  The reasoning for doing this is to use both classes as a way to enforce stricter loss on the initial Linear layer (inputs -&gt; 100 dimensional array).  Plus it seems to me it’ll be more efficient to train one network that predicts two things than two networks that each predict one thing.</p>
<p>Furthermore, given that the RNN in the tutorial shown above returns <code>output, hidden</code>, I assume I would need to modify it to return <code>output, hidden, binary_output</code> to return the binary prediction from the secondary network path.  Am I correct in assuming that?  And, how should I go about calculating the loss given that I currently call the loss function with <code>loss(output,target)</code>?  I would need to somehow incorporate the loss from the <code>binary_output</code> as well.</p>
<p>Thanks</p>

username: SimonW
cooked: <p>You can use the <code>output</code> variable to generate two outputs. There are multiple ways, e.g., something like the <code>forward</code> in your code above.</p>
<p>To backprop, just do loss on different outputs, add them together, and call backward on the sum.</p>

username: shavitamit
cooked: <p>Calculating the loss for each individual output and summing it together worked, thanks!</p>

username: FaribaKh92
cooked: <p>I have the same question. Could you please give more details on what you did? Thanks!</p>

username: shavitamit
cooked: <p>I followed <a class="mention" href="/u/simonw">@SimonW</a>’s advice and simply calculated the loss on both individual outputs.  My <code>forward</code> routine returned <code>o1</code> and <code>o2</code>.  I then calculated <code>loss1 = loss_function(o1, truth1)</code> and <code>loss2 = loss_function(o2,truth2)</code> and summed them up.  <code>loss = loss1+loss2</code>.  You can then run the backward prop on the summed-up loss.</p>

username: karl7
cooked: <p>Hi <a class="mention" href="/u/simonw">@SimonW</a> , may i ask a question?</p>
<p>If the outputs in my network come from some different layers (such as output1 comes directly from the backbone, output2 comes from the fc classification layer–just the layer after the backbone), can i also add their losses together and then backward once?</p>
<p>Or i need to backward them separately? because the losses constrain the outputs of different layers in the network…</p>
<p>Look forward to your reply, thank you!</p>

username: SimonW
cooked: <p>you can sum them together and backprop once</p>