Multidimensional BCEWithLogitsLoss

gquenot: <p>Hello,<br>
I am trying to use BCEWithLogitsLoss() with additional dimensions but it crashes. The error messages are different in cases the additional axis has a size of 1 or 2:</p>
<pre><code class="lang-auto">batch_size = 8
groups = 1
num_classes = 10
pos_weight = (num_classes-1.0)*torch.ones([num_classes])
criterion = nn.BCEWithLogitsLoss(pos_weight=pos_weight)
x = torch.rand(batch_size, num_classes, groups)
print(x.shape)
# print(x)
y = torch.empty(batch_size, dtype=torch.long).random_(num_classes)
print(y.shape)
y = y.unsqueeze(1).expand(-1,groups)
print(y.shape)
y = F.one_hot(y, num_classes).type_as(x).transpose(1, 2)
print(y.shape)
# print(y)
criterion(x,y)
</code></pre>
<pre><code class="lang-auto">torch.Size([8, 10, 1])
torch.Size([8])
torch.Size([8, 1])
torch.Size([8, 10, 1])

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-36-5314aeccd9bc&gt; in &lt;module&gt;
     14 print(y.shape)
     15 # print(y)
---&gt; 16 criterion(x,y)

/srv/storage/irim@storage1.lille.grid5000.fr/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    530             result = self._slow_forward(*input, **kwargs)
    531         else:
--&gt; 532             result = self.forward(*input, **kwargs)
    533         for hook in self._forward_hooks.values():
    534             hook_result = hook(self, input, result)

/srv/storage/irim@storage1.lille.grid5000.fr/anaconda3/lib/python3.7/site-packages/torch/nn/modules/loss.py in forward(self, input, target)
    599                                                   self.weight,
    600                                                   pos_weight=self.pos_weight,
--&gt; 601                                                   reduction=self.reduction)
    602 
    603 

/srv/storage/irim@storage1.lille.grid5000.fr/anaconda3/lib/python3.7/site-packages/torch/nn/functional.py in binary_cross_entropy_with_logits(input, target, weight, size_average, reduce, reduction, pos_weight)
   2124         raise ValueError("Target size ({}) must be the same as input size ({})".format(target.size(), input.size()))
   2125 
-&gt; 2126     return torch.binary_cross_entropy_with_logits(input, target, weight, pos_weight, reduction_enum)
   2127 
   2128 

RuntimeError: output with shape [8, 10, 1] doesn't match the broadcast shape [8, 10, 10]

</code></pre>
<pre><code class="lang-auto">batch_size = 8
groups = 2
num_classes = 10
pos_weight = (num_classes-1.0)*torch.ones([num_classes])
criterion = nn.BCEWithLogitsLoss(pos_weight=pos_weight)
x = torch.rand(batch_size, num_classes, groups)
print(x.shape)
# print(x)
y = torch.empty(batch_size, dtype=torch.long).random_(num_classes)
print(y.shape)
y = y.unsqueeze(1).expand(-1,groups)
print(y.shape)
y = F.one_hot(y, num_classes).type_as(x).transpose(1, 2)
print(y.shape)
# print(y)
criterion(x,y)
</code></pre>
<pre><code class="lang-auto">
torch.Size([8, 10, 2])
torch.Size([8])
torch.Size([8, 2])
torch.Size([8, 10, 2])

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-37-f22703bfb833&gt; in &lt;module&gt;
     14 print(y.shape)
     15 # print(y)
---&gt; 16 criterion(x,y)

/srv/storage/irim@storage1.lille.grid5000.fr/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    530             result = self._slow_forward(*input, **kwargs)
    531         else:
--&gt; 532             result = self.forward(*input, **kwargs)
    533         for hook in self._forward_hooks.values():
    534             hook_result = hook(self, input, result)

/srv/storage/irim@storage1.lille.grid5000.fr/anaconda3/lib/python3.7/site-packages/torch/nn/modules/loss.py in forward(self, input, target)
    599                                                   self.weight,
    600                                                   pos_weight=self.pos_weight,
--&gt; 601                                                   reduction=self.reduction)
    602 
    603 

/srv/storage/irim@storage1.lille.grid5000.fr/anaconda3/lib/python3.7/site-packages/torch/nn/functional.py in binary_cross_entropy_with_logits(input, target, weight, size_average, reduce, reduction, pos_weight)
   2124         raise ValueError("Target size ({}) must be the same as input size ({})".format(target.size(), input.size()))
   2125 
-&gt; 2126     return torch.binary_cross_entropy_with_logits(input, target, weight, pos_weight, reduction_enum)
   2127 
   2128 

RuntimeError: The size of tensor a (10) must match the size of tensor b (2) at non-singleton dimension 2
</code></pre>
<p>In both cases, I am unable to understand the error message. The x and y tensors have the same shape and the documentation says that there can be any number of additional dimensions after the batch one. Can anyone tell me what I got wrong?<br>
Thanks.</p>
<p>Best regards,</p>
<p>Georges Quénot.</p>

KFrank: <p>Hello Georges!</p>
<aside class="quote no-group quote-modified" data-username="gquenot" data-post="1" data-topic="79756" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/g/35a633/48.png" class="avatar"> gquenot:</div>
<blockquote>
<p>Hello,<br>
I am trying to use BCEWithLogitsLoss() with additional dimensions but it crashes. The error messages are different in cases the additional axis has a size of 1 or 2:</p>
<p>…</p>
<pre><code class="lang-auto">RuntimeError: output with shape [8, 10, 1] doesn't match the broadcast shape [8, 10, 10]
</code></pre>
<p>…</p>
<pre><code class="lang-auto">RuntimeError: The size of tensor a (10) must match the size of tensor b (2) at non-singleton dimension 2
</code></pre>
<p>In both cases, I am unable to understand the error message. The x and y tensors have the same shape and the documentation says that there can be any number of additional dimensions after the batch one. Can anyone tell me what I got wrong?</p>
</blockquote>
</aside>
<p>Your use of <code>BCEWithLogitsLoss</code> looks correct to me.  I am unable<br>
to reproduce the errors you are getting using pytorch version 0.3.0.<br>
Note. 0.3.0 does not support the <code>pos_weight</code> argument to the<br>
<code>BCEWithLogitsLoss</code> constructor, so maybe there’s a hint there.</p>
<p>Could you tell us what version of pytorch you are using?</p>
<p>Two things you might try:</p>
<p>First, try leaving out the <code>pos_weight</code> argument when you construct<br>
<code>criterion</code>:</p>
<pre data-code-wrap="python"><code class="lang-python">criterion = torch.nn.BCEWithLogitsLoss()
</code></pre>
<p>Second, does it work for you if you don’t use “additional dimensions?”<br>
That is, try it without the <code>groups</code> dimension (no <code>groups</code> at all, not just<br>
<code>groups = 1</code>).</p>
<p>Here is my version of your script, tweaked to run with pytorch 0.3.0,<br>
hence no <code>pos_weight</code> (and no built-in <code>one_hot()</code>):</p>
<pre data-code-wrap="python"><code class="lang-python">import torch
torch.__version__
torch.manual_seed (2020)

batch_size = 8
groups = 1
num_classes = 10

# pos_weight = (num_classes-1.0)*torch.ones([num_classes])
# criterion = nn.BCEWithLogitsLoss(pos_weight=pos_weight)
criterion = torch.nn.BCEWithLogitsLoss()

x = torch.rand (batch_size, num_classes, groups)
print (x.shape)

y = torch.zeros (batch_size).long().random_ (num_classes)
print (y.shape)
y = y.unsqueeze (1).expand (-1, groups)
print (y.shape)
# one-hot y
y = torch.zeros (batch_size, num_classes, groups).scatter_ (1, y.unsqueeze (1), 1.0)
print (y.shape)

print (x[0])
print (y[0])
criterion (x, y)

groups = 2

x = torch.rand (batch_size, num_classes, groups)
print (x.shape)

y = torch.zeros (batch_size).long().random_ (num_classes)
print (y.shape)
y = y.unsqueeze (1).expand (-1, groups)
print (y.shape)
# one-hot y
y = torch.zeros (batch_size, num_classes, groups).scatter_ (1, y.unsqueeze (1), 1.0)
print (y.shape)

print (x[0])
print (y[0])
criterion (x, y)
</code></pre>
<p>And here is the output:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.__version__
'0.3.0b0+591e73e'
&gt;&gt;&gt; torch.manual_seed (2020)
&lt;torch._C.Generator object at 0x000001A55AB46630&gt;
&gt;&gt;&gt;
&gt;&gt;&gt; batch_size = 8
&gt;&gt;&gt; groups = 1
&gt;&gt;&gt; num_classes = 10
&gt;&gt;&gt;
&gt;&gt;&gt; # pos_weight = (num_classes-1.0)*torch.ones([num_classes])
... # criterion = nn.BCEWithLogitsLoss(pos_weight=pos_weight)
... criterion = torch.nn.BCEWithLogitsLoss()
&gt;&gt;&gt;
&gt;&gt;&gt; x = torch.rand (batch_size, num_classes, groups)
&gt;&gt;&gt; print (x.shape)
torch.Size([8, 10, 1])
&gt;&gt;&gt;
&gt;&gt;&gt; y = torch.zeros (batch_size).long().random_ (num_classes)
&gt;&gt;&gt; print (y.shape)
torch.Size([8])
&gt;&gt;&gt; y = y.unsqueeze (1).expand (-1, groups)
&gt;&gt;&gt; print (y.shape)
torch.Size([8, 1])
&gt;&gt;&gt; # one-hot y
... y = torch.zeros (batch_size, num_classes, groups).scatter_ (1, y.unsqueeze (1), 1.0)
&gt;&gt;&gt; print (y.shape)
torch.Size([8, 10, 1])
&gt;&gt;&gt;
&gt;&gt;&gt; print (x[0])

 0.4869
 0.1052
 0.5883
 0.1161
 0.4949
 0.2824
 0.5899
 0.8105
 0.2512
 0.6307
[torch.FloatTensor of size 10x1]

&gt;&gt;&gt; print (y[0])

    1
    0
    0
    0
    0
    0
    0
    0
    0
    0
[torch.FloatTensor of size 10x1]

&gt;&gt;&gt; criterion (x, y)
0.9552074257284403
&gt;&gt;&gt;
&gt;&gt;&gt; groups = 2
&gt;&gt;&gt;
&gt;&gt;&gt; x = torch.rand (batch_size, num_classes, groups)
&gt;&gt;&gt; print (x.shape)
torch.Size([8, 10, 2])
&gt;&gt;&gt;
&gt;&gt;&gt; y = torch.zeros (batch_size).long().random_ (num_classes)
&gt;&gt;&gt; print (y.shape)
torch.Size([8])
&gt;&gt;&gt; y = y.unsqueeze (1).expand (-1, groups)
&gt;&gt;&gt; print (y.shape)
torch.Size([8, 2])
&gt;&gt;&gt; # one-hot y
... y = torch.zeros (batch_size, num_classes, groups).scatter_ (1, y.unsqueeze (1), 1.0)
&gt;&gt;&gt; print (y.shape)
torch.Size([8, 10, 2])
&gt;&gt;&gt;
&gt;&gt;&gt; print (x[0])

 0.0668  0.2524
 0.9783  0.9895
 0.0449  0.4335
 0.2581  0.3636
 0.1338  0.0855
 0.9679  0.7406
 0.4356  0.2211
 0.9607  0.8944
 0.3491  0.3937
 0.3645  0.1764
[torch.FloatTensor of size 10x2]

&gt;&gt;&gt; print (y[0])

    0     0
    0     0
    0     0
    0     0
    0     0
    0     0
    0     0
    1     1
    0     0
    0     0
[torch.FloatTensor of size 10x2]

&gt;&gt;&gt; criterion (x, y)
0.9362296428531408
</code></pre>
<p>Maybe someone could try this with <code>pos_weight</code> on pytorch 1.x (or<br>
whenever <code>pos_weight</code> was added).</p>
<p>Best.</p>
<p>K. Frank</p>

gquenot: <p>Dear K. Franck,</p>
<p>Thanks for you answer.</p>
<p>I am using pytorch version 1.4.0</p>
<p>No error happens when I don’t use <code>pos_weight</code> and an additional dimension simultaneously so it was a good guess to suspect an interaction.</p>
<p>Both:</p>
<pre><code class="lang-auto">import torch
import torch.nn.functional as F
print(torch.__version__)
batch_size = 8
groups = 2
num_classes = 10
print(pos_weight)
criterion = nn.BCEWithLogitsLoss()
x = torch.rand(batch_size, num_classes, groups)
print(x.shape)
# print(x)
y = torch.empty(batch_size, dtype=torch.long).random_(num_classes)
print(y.shape)
y = y.unsqueeze(1).expand(-1,groups)
print(y.shape)
y = F.one_hot(y, num_classes).type_as(x).transpose(1, 2)
print(y.shape)
# print(y)
criterion(x,y)
</code></pre>
<p>and</p>
<pre><code class="lang-auto">import torch
import torch.nn.functional as F
print(torch.__version__)
batch_size = 8
groups = 2
num_classes = 10
pos_weight = (num_classes-1.0)*torch.ones([num_classes])
print(pos_weight)
criterion = nn.BCEWithLogitsLoss(pos_weight=pos_weight)
x = torch.rand(batch_size, num_classes)
print(x.shape)
# print(x)
y = torch.empty(batch_size, dtype=torch.long).random_(num_classes)
print(y.shape)
y = F.one_hot(y, num_classes).type_as(x)
print(y.shape)
# print(y)
criterion(x,y)
</code></pre>
<p>work just fine. When used, <code>pos_weight</code> is:<br>
<code>tensor([9., 9., 9., 9., 9., 9., 9., 9., 9., 9.])</code><br>
which looks like what is expected by the function according to the documentation: <strong>pos_weight</strong> (<a href="https://pytorch.org/docs/stable/tensors.html#torch.Tensor" rel="nofollow noopener"> <em>Tensor</em> </a> <em>,</em> <em>optional</em> ) – a weight of positive examples. Must be a vector with length equal to the number of classes.</p>
<p>It happens that the following works:</p>
<pre><code class="lang-auto">import torch
import torch.nn.functional as F
print(torch.__version__)
batch_size = 8
groups = 2
num_classes = 10
pos_weight = (num_classes-1.0)*torch.ones([num_classes, groups])
print(pos_weight)
criterion = nn.BCEWithLogitsLoss(pos_weight=pos_weight)
x = torch.rand(batch_size, num_classes, groups)
print(x.shape)
# print(x)
y = torch.empty(batch_size, dtype=torch.long).random_(num_classes)
print(y.shape)
y = y.unsqueeze(1).expand(-1,groups)
print(y.shape)
y = F.one_hot(y, num_classes).type_as(x).transpose(1, 2)
print(y.shape)
# print(y)
print(criterion(x,y))
</code></pre>
<p>with output:</p>
<pre><code class="lang-auto">1.4.0
tensor([[9., 9.],
        [9., 9.],
        [9., 9.],
        [9., 9.],
        [9., 9.],
        [9., 9.],
        [9., 9.],
        [9., 9.],
        [9., 9.],
        [9., 9.]])
torch.Size([8, 10, 2])
torch.Size([8])
torch.Size([8, 2])
torch.Size([8, 10, 2])
tensor(1.3508)
</code></pre>
<p>So <code>pos_weight</code> should have the same size as <code>input</code> and <code>target</code> (without the batch axis) but this is not clear in the documentation.</p>
<p>Best regards,</p>
<p>Georges.</p>

KFrank: <p>Hi Georges!</p>
<aside class="quote no-group quote-modified" data-username="gquenot" data-post="3" data-topic="79756" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/g/35a633/48.png" class="avatar"> gquenot:</div>
<blockquote>
<p>It happens that the following works:<br>
…<br>
So <code>pos_weight</code> should have the same size as <code>input</code> and <code>target</code> (without the batch axis) but this is not clear in the documentation.</p>
</blockquote>
</aside>
<p>Thanks.  That makes it pretty clear what is going on.</p>
<p>I would call this a bug.  It somewhat defeats the purpose of<br>
<code>pos_weight</code>, and, in my view, breaks its semantics.  (In any<br>
event, I would say it’s at least a documentation bug.)</p>
<p>Perhaps someone like <a class="mention" href="/u/alband">@albanD</a> could take a look …</p>
<p>Best.</p>
<p>K. Frank</p>