Adding Losses of items in output array vs Loss on entire Array

AhmadMoussa: <p>I hope my title makes sense. I would like to know whether the following two snippets of code are equivalent or not:</p>
<pre><code class="lang-auto">criterion = nn.MSELoss()
output = net(input)
loss = criterion(output, target)
loss.backward()
</code></pre>
<p>Where output is an array of multiple values. Would the next snippet of code be identical?</p>
<pre><code class="lang-auto">criterion = nn.MSELoss()
output = net(input)
loss = 0
for (out, tar) in zip(output, target):
    loss += criterion(out, tar)
loss.backward()
</code></pre>
<p>Now assume I am not using MSELoss but a custom loss function I wrote myself. Would this still hold?</p>

ptrblck: <p>The result should be the same, if you use the sum as the reduction type (<code>reduction='sum'</code> for <code>nn.MSELoss</code>).<br>
Thus it depends also what reduction you are using in your custom loss function.</p>
<p>Here is an example code snippet using <code>nn.CrossEntropyLoss</code>.<br>
Note that I called <code>model.eval()</code> to get the same outputs. Otherwise the first forward pass would update the batchnorm layers, which would yield a small difference.</p>
<pre><code class="lang-python">model = models.resnet18().eval()

x = torch.randn(10, 3, 224, 224)
target = torch.randint(0, 1000, (10,))

criterion = nn.CrossEntropyLoss(reduction='sum')

output = model(x)
loss = criterion(output, target)
loss.backward()

grads1 = []
for param in model.parameters():
    grads1.append(param.grad.clone())

model.zero_grad()
output = model(x)
loss = 0
for o, t in zip(output, target):
    loss += criterion(o.unsqueeze(0), t.unsqueeze(0))
loss.backward()

grads2 = []
for param in model.parameters():
    grads2.append(param.grad.clone())


for g1, g2 in zip(grads1, grads2):
    if not torch.allclose(g1, g2):
        print('mismatch!')
</code></pre>

EGanji: <p>I have a similar question, let’s say we have X, Y and defined</p>
<pre><code class="lang-auto">noises = randn_like(X)
oprimizer = optim.Adam([noises], lr=lr, weight_decay=w_decay)
criterion = nn.CrossEntropyLoss(reduction='none')
for i in range(max_iter):
    outs = model(X + noises)
    loss = criterion(outs, targets)
    loss.backward()
    optimizer.step()
    optimizer.zero_grad()
</code></pre>
<p>here the loss is an array, but I want to apply each individual loo into each of noises corresponding to each sample of X and corresponding label in Y.<br>
Som let say, X is 100-by-784, Y is 100-by-y, Noises is 100-by-784, then loss would be 100-by-1.</p>
<p>I want the loss.backward update Noises accordingly. But it seems loss.backward only works on scalar values! that’s disappointing! There should be a way to parallelize the whole process, which I might not aware of.</p>
<p>Please help me in this matter.</p>

ptrblck: <aside class="quote no-group" data-username="EGanji" data-post="3" data-topic="67670">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/eganji/48/63367_2.png" class="avatar"> EGanji:</div>
<blockquote>
<p>I want the loss.backward update Noises accordingly. But it seems loss.backward only works on scalar values! that’s disappointing! There should be a way to parallelize the whole process, which I might not aware of.</p>
</blockquote>
</aside>
<p>That’s not the case as <code>backward</code> works on any tensors if you provide the corresponding <code>gradient</code>:</p>
<pre data-code-wrap="python"><code class="lang-python">x = torch.randn(5, 5, requires_grad=True)
x.backward(gradient=torch.ones_like(x) * 0.1)
print(x.grad)
# tensor([[0.1000, 0.1000, 0.1000, 0.1000, 0.1000],
#         [0.1000, 0.1000, 0.1000, 0.1000, 0.1000],
#         [0.1000, 0.1000, 0.1000, 0.1000, 0.1000],
#         [0.1000, 0.1000, 0.1000, 0.1000, 0.1000],
#         [0.1000, 0.1000, 0.1000, 0.1000, 0.1000]])
</code></pre>