BCE Loss Runtime Error

Achakma: <p>Hello,</p>
<p>I’m trying to train an adversarial network and I’m using BCELoss from PyTorch. I’ve provided the discriminator network, training code, error message snippet below. I’m training the network only for 5 epochs and there is no error generated for the initial 4 epochs but stuck into a runtime error at the fifth epoch. Any suggestions, please?</p>
<p>Discriminator Network</p>
<blockquote>
<p>class Discriminator(nn.Module):<br>
def <strong>init</strong>(self):<br>
super(Discriminator, self).<strong>init</strong>()<br>
self.Dconv1 = nn.Sequential(<br>
nn.Conv2d(in_channels=128, out_channels=256, kernel_size=(1, 5)),<br>
nn.LeakyReLU(negative_slope=0.2)<br>
)</p>
</blockquote>
<pre><code>    self.Dconv2 = nn.Sequential(
        nn.Conv2d(in_channels=256, out_channels=512, kernel_size=(1, 3)),
        nn.LeakyReLU(negative_slope=0.2)
    )  
    

    self.Dfc1 = nn.Sequential(
        nn.Linear(in_features=1536, out_features=256),
        nn.LeakyReLU(negative_slope=0.2),
        nn.Dropout(),
        nn.Linear(in_features=256, out_features=1),
        nn.Sigmoid()
    )
   
    self._initialize_weights()

def _initialize_weights(self):
    for m in self.modules():
        if isinstance(m, nn.Conv2d):
            n = m.kernel_size[0] * m.kernel_size[1] * m.out_channels
            m.weight.data.normal_(0, math.sqrt(2. / n))
            if m.bias is not None:
                m.bias.data.zero_()
        elif isinstance(m, nn.BatchNorm2d):
            m.weight.data.fill_(1)
            m.bias.data.zero_()
        elif isinstance(m, nn.Linear):
            n = m.weight.size(1)
            m.weight.data.normal_(0, 0.01)
            m.bias.data.zero_()

def forward(self, input):
    out = self.Dconv1(input)
    out = self.Dconv2(out)
    out = out.reshape(-1, 512*1*3)
    out = self.Dfc1(out)
    return out
</code></pre>
<p>Training Code Snippet</p>
<blockquote></blockquote>
<pre><code>        s1_length = len(s1_source)
        s2_length = len(s2_source)
        t_length= len(s1_target)
        logging.warning("Iteration: %d, S1 length: %d, S2 length: %d, Target length: %d", i, s1_length, s2_length, t_length)
        
        s1_error_fake = loss(s1_source, ones_target(s1_length))
        s1_error_real = loss(s1_target, zeros_target(t_length))
        s1_t_dis_loss = s1_error_fake + s1_error_real
        
        s2_error_fake = loss(s2_source, ones_target(s2_length))
        s2_error_real = loss(s2_target, zeros_target(t_length))
        s2_t_dis_loss = s2_error_fake + s2_error_real
        
        logging.warning("S1 Disc loss: %s, S2 Disc Loss: %s", s1_t_dis_loss.data, s2_t_dis_loss.data)
</code></pre>
<p>Error Message</p>
<blockquote>
<p>146it [00:02, 53.81it/s]<br>
146it [00:02, 56.52it/s]<br>
146it [00:02, 57.55it/s]<br>
42it [00:00, 58.23it/s]</p>
</blockquote>
<hr>
<p>RuntimeError                              Traceback (most recent call last)<br>
 in <br>
63             logging.warning(“Iteration: %d, S1 length: %d, S2 length: %d, Target length: %d”, i, s1_length, s2_length, t_length)<br>
64<br>
—&gt; 65             s1_error_fake = loss(s1_source, ones_target(s1_length))<br>
66             s1_error_real = loss(s1_target, zeros_target(t_length))<br>
67             s1_t_dis_loss = s1_error_fake + s1_error_real</p>
<p>~/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py in <strong>call</strong>(self, *input, **kwargs)<br>
491             result = self._slow_forward(*input, **kwargs)<br>
492         else:<br>
 → 493             result = self.forward(*input, **kwargs)<br>
494         for hook in self._forward_hooks.values():<br>
495             hook_result = hook(self, input, result)</p>
<p>~/anaconda3/lib/python3.7/site-packages/torch/nn/modules/loss.py in forward(self, input, target)<br>
510     <span class="mention">@weak_script_method</span><br>
511     def forward(self, input, target):<br>
 → 512         return F.binary_cross_entropy(input, target, weight=self.weight, reduction=self.reduction)<br>
513<br>
514</p>
<p>~/anaconda3/lib/python3.7/site-packages/torch/nn/functional.py in binary_cross_entropy(input, target, weight, size_average, reduce, reduction)<br>
2111<br>
2112     return torch._C._nn.binary_cross_entropy(<br>
 → 2113         input, target, weight, reduction_enum)<br>
2114<br>
2115</p>
<p>RuntimeError: reduce failed to synchronize: device-side assert triggered</p>
<p>1</p>
<p>​</p>

AleksandraDeis: <p>Hi! May be this helps:<br>
<a href="https://github.com/pytorch/pytorch/issues/5560" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/5560</a></p>
<p>You can also try to run your code on CPU, you will receive much more informative error message then.</p>

Achakma: <p>Thank you for the link. I actually read through that link yesterday and as you can see in my discriminator class I’ve used a Sigmoid at the final output. I also used the target tensors using the following code snippet -</p>
<p>def ones_target(size):<br>
‘’’<br>
Tensor containing ones, with shape = size<br>
‘’’<br>
data = Variable(torch.ones(size, 1)).cuda(gpu_id)<br>
return data</p>
<p>def zeros_target(size):<br>
‘’’<br>
Tensor containing zeros, with shape = size<br>
‘’’<br>
data = Variable(torch.zeros(size, 1)).cuda(gpu_id)<br>
return data</p>
<p>I’m pretty sure, I’ve done all the fixes that link suggests. I’m now confused about what could be wrong. What really annoying is that the error generates after running few epochs!</p>

AleksandraDeis: <p>Yes, it is confusing.<br>
Did you try to print out the loss? May be it is goes to infinity or vanishes causing the overflow?</p>