Cross Entropy Loss vs Batch Size

username: maralm
cooked: <p>Is that normal that cross entropy loss is increasing by increasing the batch size?<br>
I have the following loss:</p>
<pre><code class="lang-auto">loss_fct = CrossEntropyLoss()
loss = loss_fct(logits.view(-1, self.num_labels), labels.view(-1))
</code></pre>
<p>I am comparing the batch size of 32 using two methods:<br>
1- Using device batch size=32<br>
2- Using device batch size=2 with gradient accumulation step=16</p>
<p>For the first approach, loss starts fro, 0.6599792838096619 while for the second approach it starts from 0.0303945392370224. It looks like it is scaled by the batch size. Do I need to divide it by batch size again before backward or it is correct?</p>

username: KFrank
cooked: <p>Hello Maral!</p>
<aside class="quote no-group" data-username="maralm" data-post="1" data-topic="72493" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/e36b37/48.png" class="avatar"> maralm:</div>
<blockquote>
<p>Is that normal that cross entropy loss is increasing by increasing the batch size?</p>
</blockquote>
</aside>
<p>By default, <code>CrossEntropyLoss</code> does not increase with batch size.</p>
<p>From its <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.CrossEntropyLoss" rel="noopener nofollow ugc">documentation</a>, unless you explicitly construct it with<br>
<code>reduction = 'sum'</code>, it will default to <code>reduction = 'mean'</code>,<br>
for which “the sum of the output will be divided by the number<br>
of elements in the output.”</p>
<p>Here is a short (pytorch version 0.3.0) script that illustrates this:</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
torch.__version__

torch.manual_seed (2020)

pred2 = torch.autograd.Variable (torch.randn (2, 5))
targ2 = torch.autograd.Variable (torch.LongTensor (2).random_ (0, 5))

pred2
targ2

pred32 = pred2.repeat (16, 1)
targ32 = targ2.repeat (16)

pred32.shape
targ32.shape

loss_fn = torch.nn.CrossEntropyLoss()

loss_fn (pred2, targ2)
loss_fn (pred32, targ32)
</code></pre>
<p>And here is the output:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'0.3.0b0+591e73e'
&gt;&gt;&gt;
&gt;&gt;&gt; torch.manual_seed (2020)
&lt;torch._C.Generator object at 0x0000020A70856630&gt;
&gt;&gt;&gt;
&gt;&gt;&gt; pred2 = torch.autograd.Variable (torch.randn (2, 5))
&gt;&gt;&gt; targ2 = torch.autograd.Variable (torch.LongTensor (2).random_ (0, 5))
&gt;&gt;&gt;
&gt;&gt;&gt; pred2
Variable containing:
 1.2372 -0.9604  1.5415 -0.4079  0.8806
 0.0529  0.0751  0.4777 -0.6759 -2.1489
[torch.FloatTensor of size 2x5]

&gt;&gt;&gt; targ2
Variable containing:
 0
 1
[torch.LongTensor of size 2]

&gt;&gt;&gt;
&gt;&gt;&gt; pred32 = pred2.repeat (16, 1)
&gt;&gt;&gt; targ32 = targ2.repeat (16)
&gt;&gt;&gt;
&gt;&gt;&gt; pred32.shape
torch.Size([32, 5])
&gt;&gt;&gt; targ32.shape
torch.Size([32])
&gt;&gt;&gt;
&gt;&gt;&gt; loss_fn = torch.nn.CrossEntropyLoss()
&gt;&gt;&gt;
&gt;&gt;&gt; loss_fn (pred2, targ2)
Variable containing:
 1.3058
[torch.FloatTensor of size 1]

&gt;&gt;&gt; loss_fn (pred32, targ32)
Variable containing:
 1.3058
[torch.FloatTensor of size 1]
</code></pre>
<aside class="quote no-group">
<blockquote>
<p>I am comparing the batch size of 32 using two methods:<br>
1- Using device batch size=32<br>
2- Using device batch size=2 with gradient accumulation step=16</p>
</blockquote>
</aside>
<p>As noted above, the larger loss is not coming from the larger batch<br>
size fed to <code>CrossEntropyLoss</code>.   Without seeing your actual code,<br>
especially how you implement</p>
<blockquote>
<p>gradient accumulation step=16,</p>
</blockquote>
<p>it’s hard to guess what might be going on.</p>
<p>Best.</p>
<p>K. Frank</p>

username: maralm
cooked: <p>Thanks! I was expecting what you said but see scaled loss for some reason. Accuracy is normal but loss scale is different for approach 1 and 2!</p>