A model with multiple outputs

Vijay_Dubey: <p>I have to implement a Convolutional Neural Network, that takes a kinect image (1<em>640</em>480) and return a 1 x8 tensor predicting the class to which the object belongs and a 1 x 4 tensor, predicting the bounding box around the image, if its present.</p>
<p>Please help me how can I implement a suitable model to give two outputs and how to calculate loss and backpropagate in that case?<br>
Also, i have just around 6000 training images, how can I achieve the best possible results with limited number of training images??</p>
<p>Some examples of such projects would be highly helpful.</p>

ptrblck: <p>Multiple outputs is pretty straightforward. Just return mutiple values in the <code>forward()</code> method of your net.</p>
<pre><code>def forward(self, x):
    # Do your stuff here
    ...
    x1 = F.log_softmax(x) # class probabilities
    x2 = ... # bounding box calculation
    return x1, x2
</code></pre>
<p>Using these two outputs, you can define two different loss functions and just add them.</p>
<pre><code>out1, out2 = model(data)
loss1 = criterion1(out1, target1)
loss2 = criterion2(out2, target2)
loss = loss1 + loss2
loss.backward()</code></pre>

smth: <p>please dont tag folks generically, it’s not helpful</p>

Vijay_Dubey: <p>Ok<br>
I am sorry for that<br>
Thanks</p>

Pt_S: <p>Thanks for your answer!</p>

xiaoxifei1223: <p>Hi, Smth<br>
Actually , I still don<code>t understand your idea about tag folks, does it mean the forward function from ptrblck</code>s example sets tow return value or his loss function uses tow loss addition ？ can you explain it in detail ？ can you give a example code ？<br>
Thank you very much</p>

jia_lee: <p>Hi, can we pack the returned tensors in list format? Such as:<br>
list = [x1, x2]<br>
return list</p>
<p>Or, we must return  tensor class type in forward method?</p>

ptrblck: <p>Yes, you can also return a list. Here is a small dummy example:</p>
<pre><code class="lang-python">class MyModel(nn.Module):
    def __init__(self):
        super(MyModel, self).__init__()
        self.fc1 = nn.Linear(1, 1)
        
    def forward(self, x):
        y = self.fc1(x)
        return [y, x]

model = MyModel()
x = torch.randn(1, 1)
outputs = model(x)
print(outputs)
</code></pre>

jia_lee: <p>Thank you so much for your quick help!</p>

Shivam_Chandhok: <p>Hey,<br>
I was going through the keras documentation<br>
<a href="https://keras.io/getting-started/functional-api-guide/#multi-input-and-multi-output-models" class="onebox" target="_blank" rel="nofollow noopener">https://keras.io/getting-started/functional-api-guide/#multi-input-and-multi-output-models</a><br>
As you can see this is an example of multi-output multi input model.The thing in this example is that the auxiliary output has a loss function .This output is then further used in the model to compute the main_output.<br>
How can I implement some thing like this in Pytorch</p>

ptrblck: <p>You could just re-write the forward pass in your model similar to this:</p>
<pre><code class="lang-python">def forward(self, main_input, aux_input):
    main_input = self.embedding_1(main_input)
    main_input = self.lstm(main_input, (self.hidden, self.state))
    aux = self.aux_output(main_input)    

    aux_input = self.input_layer(aux_input)
    x = torch.cat((aux_input, main_input))
    x = F.relu(self.dense_1(x))
    x = F.relu(self.dense_2(x))
    x = F.relu(self.dense_3(x))
    x = self.main_output(x)

    return x, aux
</code></pre>

Shivam_Chandhok: <p>Oh okay .Thanks a lot.<br>
Can I also use two different loss functions one on auxiliary_output and one on main_output as shown in the keras link and then add them in the end.</p>

ptrblck: <p>Sure, that should also work!<br>
Let me know, if you encounter any problems. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

hashbrown: <p>I would like to do the same, but accumulate loss in a loop as my model’s forward function returns a list of outputs.<br>
My question is, how do I initialize the “loss” variable so that I can accumulate in a loop?</p>

ptrblck: <p>If your model returns multiple outputs, you could calculate the loss separately for each output and then just accumulate it:</p>
<pre><code class="lang-python">output1, output2, output3 = model(data)
loss1 = criterion1(output1, target1)
loss2 = criterion2(output2, target2)
loss3 = criterion3(output3, target3)
loss = loss1 + loss2 + loss3

# or
outputs = model(data)
loss = 0
for output, target, criterion in zip(outputs, targets, criteria):
      loss = loss + criterion(output, target)
</code></pre>

Hazel: <p>Little confused about the accumulation “+”:<br>
So when I write loss = loss1 + loss2, the network knows that I want loss1 to backpropagate through output1, and loss2 to go thru output2, the sequence matters, am I right?</p>

ptrblck: <p>The addition is treated as any other operation in your model, so Autograd will calculate the gradients based on the derivative of the loss accumulation.</p>
<p>If the outputs are independent, your assumption is correct.</p>

BESMA_KHALFOUN: <p>hey , i have a problem with “Assertion `cur_target &gt;= 0 &amp;&amp; cur_target &lt; n_classes’ failed.  at /pytorch/aten/src/THNN/generic/ClassNLLCriterion.c:97” which is still persisting even i tried many solutions found on internet.</p>

ptrblck: <p>Could you print the target values of your current batch inside the training loop and check, if they are all in the range <code>[0, nb_classes-1]</code>?<br>
If you run the code with <code>num_workers=0</code>, the last printed target should be the one causing this error.</p>

BESMA_KHALFOUN: <p>It start counting from 1.</p>