Multiple Losses and Optimization + Weight Regularization

username: tsuijenk
cooked: <p>Hello everyone,</p>
<p>There doesn’t seem to be a lot of existing literatures that I found regarding one loss vs. multiple losses for single input.</p>
<p>One of two outputs, namely “Output 2”, for my architecture is a stack of N images.</p>
<p>Right now, I use a single loss function to “control” this “Output2” (which is a stack of N images). And I have tested several different options. One example is calculating the sub-loss for each of N images, and then take “reduction=‘mean’”, hence returning a single <strong>loss2</strong> for each batch.</p>
<p>So currently, I do the following:</p>
<pre><code class="lang-auto">out = network(x)
loss = list(map(out, x)) #same as a for-loop to loop over 32 batches
mean_loss = loss/32 # loss.shape == (32, 2)
loss1 = loss[:,0].mean() #for output1
loss2 = loss[:,1].mean() #for output2
</code></pre>
<p>But I have been wondering if the model would back propagate differently if I explicitly define Loss2 as multiple N losses, thus increasing accuracy?</p>
<pre><code class="lang-auto">out = network(x)
loss = list(map(out, x)) #same as a for-loop to loop over 32 batches
mean_loss = loss/32 # loss.shape == (32,10)
loss1 = loss[:,0].mean() #for output1
loss2 = loss[:,1].mean() #for output2 img 1
loss3 = loss[:,2].mean() # for output2 img 2
...
loss 10 = loss[:,10].mean() for #output2 img 10
</code></pre>
<p>Thank you!</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="tsuijenk" data-post="1" data-topic="143847">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tsuijenk/48/45140_2.png" class="avatar"> tsuijenk:</div>
<blockquote>
<p>if I explicitly define Loss2 as multiple N losses, thus increasing accuracy?</p>
</blockquote>
</aside>
<p>I’m not sure how to interpret the “increase accuracy” part so could you explain it a bit more?<br>
Generally, calling <code>backward</code> on a reduced loss vs. indexing the loss first would yield the same gradients if you include the gradient accumulation (you could scale the indexed losses down) as seen here:</p>
<pre data-code-wrap="python"><code class="lang-python">model = models.resnet18()
model.eval()
criterion = nn.CrossEntropyLoss(reduction='none')

x = torch.randn(3, 3, 224, 224)
target = torch.randint(0, 1000, (3,))

# single loss
out = model(x)
loss = criterion(out, target)
loss.mean().backward()

print(model.conv1.weight.grad.abs().sum())
# &gt; tensor(369.1813)

# loss indexing
model.zero_grad()
out = model(x)
loss = criterion(out, target)
for idx in range(loss.size(0)):
    loss[idx].backward(retain_graph=True)
print(model.conv1.weight.grad.abs().sum() / x.size(0))
# &gt; tensor(369.1814)
</code></pre>

username: tsuijenk
cooked: <p>Hello! Thanks for your reply and suggestion.</p>
<p>I mentioned “increasing accuracy”, because the output1 is a single image and uses a variant of Cross Entropy and it works perfectly, high jaccard index, etc.</p>
<p>Hence, my intuition is that for output2 (which is a stack of N images), I can think of it merging the stack of N images as a single image and fit a similar variant of Cross Entropy (like the one used for output1). But the jaccard index is low, lots of false positives.</p>
<p>So this makes me wonder if treating output2 (the stack of N images) using N cross entropy loss individually would improve the jaccard index.</p>
<p>I hope this clarifies my thoughts a bit more! Thank you!</p>

username: tsuijenk
cooked: <p>Hello! I think you may have answer my question indirectly then actually.</p>
<p>Suppose output2 loss = mean(subloss1, subloss2, subloss3, …, sublossN).</p>
<p>Backward propagation of output2 loss would yield the same gradient as backward propagation of (subloss1, subloss2, subloss3, …, sublossN).</p>
<p>Is that what you meant? Thank you.</p>