Strange shape mismatch issue with torch.BCEWithLogitsLoss()

username: 4158ndfkvBHJ1
cooked: <p>I am writing a relativistic GAN. I have the following generator update function</p>
<pre><code class="lang-auto">def optimize_generator(self, real_batch, fake_batch):

        """
        relativistic generator update step
        """

        valid = torch.ones(real_batch.size(0), device = self.device)
        fake = torch.zeros(fake_batch.size(0), device = self.device)


        y_pred = self.netD(real_batch)
        y_pred_fake = self.netD(fake_batch.detach())
        
        print("(y_pred- torch.mean(y_pred_fake)).squeeze().size() : ", (y_pred- torch.mean(y_pred_fake)).squeeze().size())
        print("(y_pred_fake - torch.mean(y_pred)).squeeze().size() : ", (y_pred_fake - torch.mean(y_pred)).squeeze().size())
        print("fake.size() : ", fake.size())
        print("valid.size() : ", valid.size())

        print((y_pred- torch.mean(y_pred_fake)).squeeze().size() == fake.size())
        print((y_pred_fake - torch.mean(y_pred)).squeeze().size() == valid.size())

        real_v_fake = self.loss((y_pred- torch.mean(y_pred_fake)).squeeze(), fake)
        fake_v_real = self.loss((y_pred_fake - torch.mean(y_pred)).squeeze(), valid)
        
        g_loss = (real_v_fake + fake_v_real)/2

        """
        gradient update pass
        """

        self.gen_optim.zero_grad()
        g_loss.backward()
        self.gen_optim.step()

        return g_loss
</code></pre>
<p>I have a very strange issue with the dimensions of the input and the target, linked in the following console output :<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/b/ebe523496ae47344731e8fbd0f4cf2ec4fdefd09.png" data-download-href="https://discuss.pytorch.org/uploads/default/ebe523496ae47344731e8fbd0f4cf2ec4fdefd09" title="weird_thingie"><img src="https://discuss.pytorch.org/uploads/default/original/3X/e/b/ebe523496ae47344731e8fbd0f4cf2ec4fdefd09.png" alt="weird_thingie" data-base62-sha1="xEP87ZxoXh9h2awn4QN79d4B57b" width="690" height="212" data-dominant-color="282828"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">weird_thingie</span><span class="informations">1170×361 20.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
as you can see, the size of the input and target is the same torch.size([64]) and a boolean check of equality even succeeds, but BCEWithLogitsLoss keeps throwing this error. I had a look at the torch/nn/functional.py code, and I seem to be failing this exact check (line 2433) :</p>
<pre><code class="lang-auto">if not (target.size() == input.size()):
    raise ValueError("Target size ({}) must be the same as input size ({})".format(target.size(), input.size()))
</code></pre>
<p>I am at a bit of a loss, because I don’t see how that’s different from the boolean equality I tested before the call to BCEWithLogitsLoss. Any help would be much appreciated <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"> !</p>

username: 4158ndfkvBHJ1
cooked: <p>The equality even fails after a reshape</p>
<pre><code>    valid = torch.ones(real_batch.size(0), device = self.device)
    fake = torch.zeros(fake_batch.size(0), device = self.device)


    y_pred = self.netD(real_batch)
    y_pred_fake = self.netD(fake_batch.detach())
    
    real_v_fake = (y_pred- torch.mean(y_pred_fake))
    fake_v_real = (y_pred_fake - torch.mean(y_pred))

    real_v_fake = torch.reshape(real_v_fake,(real_batch.size(0),1))
    fake_v_real = torch.reshape(fake_v_real,(real_batch.size(0),1))
    valid = torch.reshape(valid, (real_batch.size(0),1))
    fake = torch.reshape(fake, (real_batch.size(0),1))
    
    real_v_fake = self.loss(real_v_fake, fake)
    fake_v_real = self.loss(fake_v_real, valid)
</code></pre>
<p>meaning torch.nn.BCEWithLogitsLoss still complains about different shapes even after pytorch successfully reshapes them to the exact same shape (real_batch.size(0),1).</p>

username: ptrblck
cooked: <p>Could you post an executable code snippet to reproduce this error, please?</p>

username: andre_dourson
cooked: <p>So I also have a strange issue with BCEWithLogitsLoss , simply with computation of the loss thru</p>
<p>loss = criterion(outputs.squeeze(), labels)</p>
<p>The computation works just fine 99.9% of the time but when I use this on my dataset with 7987 rows AND with a batch size of 256, I have the error</p>
<p>File “/nics/b/home/doursand/anaconda3/envs/limsnet/lib/python3.7/site-packages/torch/nn/functional.py”, line 2580, in binary_cross_entropy_with_logits<br>
raise ValueError(“Target size ({}) must be the same as input size ({})”.format(target.size(), input.size()))<br>
ValueError: Target size (torch.Size([1])) must be the same as input size (torch.Size([]))</p>
<p>However, if I change the batch size value to <strong>255</strong> instead of <strong>256</strong>, then everything works like a charm!</p>
<p>Now if you do the calculation,</p>
<p><strong>7937 rows / 256  = 31.<strong>00</strong>39062</strong></p>
<p>and</p>
<p><strong>7937 rows / 255 = 31.133333</strong></p>
<p>Question : is it a rounding issue which makes the iteration with 256 images/batch fail (as 31.0039062 is seen as 31 ?)</p>

username: 4158ndfkvBHJ1
cooked: <p>I’m an idiot, the error is not happening in optimize_generator, but in optimize_discriminator, which I called earlier. The equality in shapes succeeded … just not in the function outputting the error.</p>

username: 4158ndfkvBHJ1
cooked: <p>this is probably related to the last batch being incomplete. You can change the batch size as you did, or you can use drop_last = True as an option in the DataLoader to avoid this issue.</p>