UserWarning: size_average and reduce args will be deprecated, please use reduction='sum' instead

nullgeppetto: <p>Since upgrading PyTorch (to ‘0.5.0a0+a24163a’), I am getting the following warning:</p>
<pre><code class="lang-auto">/usr/lib/python3.7/site-packages/torch/nn/functional.py:52: UserWarning: size_average and reduce args will be deprecated, please use reduction='sum' instead.
  warnings.warn(warning.format(ret))
</code></pre>
<p>Is this due to the new pytorch version or is it due to python-3.7? Is there anything I need to do or is it going to be fixed in newer versions?</p>
<p>Thank you.</p>

ptrblck: <p>The <code>reduction</code> argument was recently introduced in <code>0.4.1</code>.<br>
See e.g. the <a href="https://pytorch.org/docs/stable/nn.html#torch.nn.CrossEntropyLoss">docs for nn.CrossEntropyLoss</a>.</p>
<p>It’s currently just a warning that you are using deprecated arguments, so you should definitely use <code>reduction</code> instead of <code>reduce</code> or <code>size_average</code>.</p>

nullgeppetto: <p>Thank you very much!</p>

kumarnikhil936: <p>You need to replace the deprecated arguments with the given suggestion.</p>
<p>See here how they are are generating the warning message.<br>
<a href="https://fossies.org/linux/pytorch/torch/nn/_reduction.py" class="onebox" target="_blank" rel="nofollow noopener">https://fossies.org/linux/pytorch/torch/nn/_reduction.py</a></p>
<p>It will also help you understand why the <code>reduction</code> argument makes the other two redundant.</p>

jphoward: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="24629">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>It’s currently just a warning that you are using deprecated arguments, so you should definitely use <code>reduction</code> instead of <code>reduce</code> or <code>size_average</code> .</p>
</blockquote>
</aside>
<p><a class="mention" href="/u/ptrblck">@ptrblck</a> the docs still say <code>size_average</code> is required  in order to use <code>ignore_index</code> in <code>F.cross_entropy</code>. It’s not clear how <code>size_average</code> is replaced by <code>reduction</code> in this case. How are we meant to use <code>ignore_index</code>?</p>

ptrblck: <p>Yeah, the documentation might be a bit misleading, as <code>size_average</code> is mentioned, while this argument is deprecated and reduction should be used instead.</p>
<p>There are basically the three reduction types, i.e. <code>'none'</code>, <code>'sum'</code>, and <code>'mean'</code>.<br>
<code>ignore_index</code> will be applied as:</p>
<ol>
<li>
<code>reduction='none'</code>: the loss will not be reduced, so you will get a loss tensor in the shape <code>[batch_size]</code>. The entries, where <code>target==ignore_index</code>, will have a zero loss.</li>
<li>
<code>reduction='mean'</code>: the reduced loss will be the average of all entries, where <code>target!=ignore_index</code>.</li>
<li>
<code>reduction='sum'</code>: the reduced loss will be the sum of the “raw loss”. Since the samples with ignored targets will get a zero loss, the sum should not change, if you filter them out or just sum over all values.</li>
</ol>
<p>Here is a small code snippet to demonstrate my understanding:</p>
<pre><code class="lang-python"># Setup
output = torch.randn(10, 10, requires_grad=True)
target = torch.arange(10)

# sanity check for plain loss without ignore_index
loss_raw = F.cross_entropy(output, target, reduction='none')

loss_mean = F.cross_entropy(output, target, reduction='mean')
print(loss_raw.mean() == loss_mean)

loss_sum = F.cross_entropy(output, target, reduction='sum')
print(loss_raw.sum() == loss_sum)

# Case 2: ignore_index=0, reduction='mean'
loss_raw_ignore = F.cross_entropy(
    output, target, reduction='none', ignore_index=0)

loss_mean_ignore = F.cross_entropy(
    output, target, reduction='mean', ignore_index=0)
print(loss_mean_ignore == loss_raw_ignore[loss_raw_ignore!=0].mean())

# Check gradients
output.grad = None
loss_mean_ignore.backward()
g0 = output.grad.clone()

output.grad = None
loss_raw_ignore[loss_raw_ignore!=0].mean().backward(retain_graph=True)
g1 = output.grad.clone()
print((g0 == g1).all())

# Case 3: ignore_index=0, reduction='sum'
loss_sum_ignore = F.cross_entropy(
    output, target, reduction='sum', ignore_index=0)
print(loss_sum_ignore == loss_raw_ignore.sum())

# Check gradients
output.grad = None
loss_sum_ignore.backward()
g0 = output.grad.clone()

output.grad = None
loss_raw_ignore.sum().backward(retain_graph=True)
g1 = output.grad.clone()

output.grad = None
loss_raw_ignore[loss_raw_ignore!=0].sum().backward()
g2 = output.grad.clone()
print((g0 == g1).all() and (g0 == g2).all())
</code></pre>
<p>Let me know, if I’m missing something.</p>

jphoward: <p>Thank you that’s great info! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>