Calculating loss for sub-networks

ayadav01: <p>I have a CNN network that consists of multiple small <code>Sequential</code> modules and each of the cnn modules have different output neurons, which i return in the end. Something like this:</p>
<pre><code class="lang-auto">class CNN:
def __init__(self):
self.cnn_module = nn.Sequential() # contains CNN layers
self.network1 = nn.Sequential() # Dense layers
self.network2 = nn.Sequential() # Dense layers
self.network3 = nn.Sequential() # Dense layers

def forward(self, x):
x = self.cnn_module(x)
x = x.flatten(batch_size, -1)
out_1 = self.network1(x)
out_2 = self.network2(x)
out_3 = self.network3(x)
return [out_1, out_2, out_3]
</code></pre>
<p>I was confused on how to calculated loss of each of the networks and then backpropogate the losses together. Would something like this work:</p>
<pre><code class="lang-auto">criterion = nn.CrossEntropyLoss()
model = CNN()
output = model(images)
loss_1 = criterion(output[0], label_0)
loss_2 = criterion(output[1], labels_1)
loss_3 = criterion(output[2], labels_2)
loss = loss_1+loss_2+loss_3
loss.backward()
optimizer.step()
</code></pre>

harsha_g: <p>What is <code>loss_4</code>? I don’t see any reason why this shouldn’t work? Did you give it a go?</p>

harsha_g: <p>If I am seeing this right, what you are doing is equivalent to concatenating all outputs and running a <code>CrossEntropyLoss</code> against the concatenated labels.</p>

ayadav01: <p>so if that’s correct, how can i pass weighted values to the loss function for each of the sub-network, since the labels are different for each of the sub-networks. Do i have to call <code>nn.CrossEntropy(weights)</code> each time when i calculate the loss ? so something like this:</p>
<pre><code class="lang-auto">criterion = nn.CrossEntropyLoss(weights_label_0)
loss_1 = criterion(output[0], label_0)

criterion = nn.CrossEntropyLoss(weights_label_1)
loss_2 = criterion(output[1], label_1)

criterion = nn.CrossEntropyLoss(weights_label_2)
loss_3 = criterion(output[2], label_2)

loss = loss_1+loss_2+loss_3
loss.backward()
optimizer.step()
</code></pre>
<p>will calling <code>nn.CrossEntropyLoss()</code> every time before calculating loss will mess up the backpropogation somehow?</p>

harsha_g: <p>Why call each time? Just once will suffice <em>à la</em></p>
<pre><code class="lang-auto">criterion1 = nn.CrossEntropyLoss(weights_label_0)
criterion2 = nn.CrossEntropyLoss(weights_label_1)
criterion3 = nn.CrossEntropyLoss(weights_label_2)

loss_1 = criterion1(output[0], label_0)
loss_2 = criterion2(output[1], label_1)
loss_3 = criterion3(output[2], label_2)
</code></pre>
<aside class="quote no-group" data-username="ayadav01" data-post="5" data-topic="86655">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/c67d28/48.png" class="avatar"> ayadav01:</div>
<blockquote>
<p>will calling <code>nn.CrossEntropyLoss()</code> every time before calculating loss will mess up the backpropogation somehow?</p>
</blockquote>
</aside>
<p>No, it will not.</p>

ayadav01: <p>should the weights vector for each class be calculated per batch or per dataset ?</p>