Multiple output tutorial/examples

ducminhkhoi: <p>I am doing with a network with multiple outputs. I have read this topic: <a href="https://discuss.pytorch.org/t/how-to-do-backward-for-a-net-with-multiple-outputs/87">How to do backward() for a net with multiple outputs?</a>. But I still don’t understand how to do it. Do you guys have any clear tutorials/examples how to do it?</p>
<p>Thanks,</p>

smth: <p>This comment has an example: <a href="https://discuss.pytorch.org/t/multi-label-classification-in-pytorch/905/11?u=smth">Multi Label Classification in pytorch</a></p>
<p>That thread has a lot of detail.</p>

ducminhkhoi: <p>Thanks for your reply. But it is not exactly what I need. My network generates 3 outputs which have different loss functions (MSE, Cross Entropy,…). Or use the same loss function but completely different outputs (e.g. classification from list 1, list 2 and list 3)</p>
<p>Like this one (where Dense = Linear)<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/1X/0e2c62df4cef75c73f46d0a3d72ebafb6fb4ecad.png" data-download-href="https://discuss.pytorch.org/uploads/default/0e2c62df4cef75c73f46d0a3d72ebafb6fb4ecad" title="multi-input-multi-output-graph.png"><img src="https://discuss.pytorch.org/uploads/default/optimized/1X/0e2c62df4cef75c73f46d0a3d72ebafb6fb4ecad_2_363x500.png" width="363" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/1X/0e2c62df4cef75c73f46d0a3d72ebafb6fb4ecad_2_363x500.png, https://discuss.pytorch.org/uploads/default/original/1X/0e2c62df4cef75c73f46d0a3d72ebafb6fb4ecad.png 1.5x, https://discuss.pytorch.org/uploads/default/original/1X/0e2c62df4cef75c73f46d0a3d72ebafb6fb4ecad.png 2x" data-dominant-color="F7F7F7"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">multi-input-multi-output-graph.png</span><span class="informations">538×741 27.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

smth: <p>doing multiple output + losses should be straight-forward. In your <code>forward</code> function of the model, you have the variables corresponding to each output, and send them through three separate loss functions.</p>
<p>The outputs of the loss functions can be backpropagated all together using <code>torch.autograd.backward</code></p>

platero: <p>Thanks for your hint. I also find <a href="https://discuss.pytorch.org/t/how-to-do-backward-for-a-net-with-multiple-outputs/87/2">this post</a>. I try to compute a loss for each output and append them to a list as the following code,</p>
<pre><code class="lang-auto">loss_seq = []
        for o in output:
            cur_loss = criterion(o, target_var)
            loss_seq.append(cur_loss)
</code></pre>
<p>Then I print the losses, which seem quite correct.</p>
<p>Then I tried to do the backpropagation</p>
<pre><code class="lang-auto"> torch.autograd.backward(loss_seq)
</code></pre>
<p>However, an error occured:</p>
<pre><code class="lang-auto"> File "example/main.py", line 174, in train
    torch.autograd.backward(loss_seq)
TypeError: backward() takes at least 2 arguments (1 given)
</code></pre>
<p>Am I doing wrong? How can I do backpropagation with mutiple loss?</p>

smth: <p>you have to give: <code>torch.autograd.backward(loss_seq, grad_seq)</code> where <code>grad_seq</code> is the gradients for each of the losses.</p>
<p>If you want you can give gradients to be all ones:<br>
<code>grad_seq = [loss_seq[0].new(1).fill_(1) for _ in range(len(loss_seq))]</code></p>

platero: <p>Thank you very much. But why can we use all ones as gradients here?</p>

smth: <p>if the loss is a scalar value, then all ones as gradients are what you will use if you call <code>loss.backward()</code>.</p>

cerisara: <p>Thanks: I understand the ones are the gradients dloss/dloss=1 that are put at the very end of the backpropagation graph.</p>
<p>But creating this grad_seq this way fails with<br>
AttributeError: ‘Variable’ object has no attribute ‘new’</p>
<p>EDIT: OK, it looks like it automatically converts tensors to variables, so this seems to work:</p>
<pre><code class="lang-auto">grad_seq = [loss_seq[0].data.new(1).fill_(1) for _ in range(len(loss_seq))]
</code></pre>

Timos_Korres: <p>Can a model output multiple outputs (as in the above experiments) and only have a single loss function?</p>