RuntimeError: CUDA error: device-side assert triggered - 1D CVAE

username: chaslie
cooked: <p>Hi,</p>
<p>using a 1D CVAE with a BCELoss function gives me the following (epoch, total number of batches, loss):</p>
<pre><code class="lang-auto">37 6912 tensor(318.8038, device='cuda:0', grad_fn=&lt;NegBackward&gt;)
38 6912 tensor(348.9748, device='cuda:0', grad_fn=&lt;NegBackward&gt;)
Traceback (most recent call last):
    data = data.cuda()

RuntimeError: CUDA error: device-side assert triggered
</code></pre>
<p>Does anyone know what is causing this?</p>
<p>The data loader statement is:</p>
<pre><code class="lang-auto">train_dataset2 = TensorDataset(X_train_2, y_train)
train_loader2 = DataLoader(train_dataset2, batch_size=BATCHSIZE, shuffle=True)
</code></pre>
<p>where X_train is the input 1d Vector and y_train is a linear array going from 1 to x where x is the length of the X_train array.</p>
<p>Chaslie</p>

username: ptrblck
cooked: <p>Could you run the code with <code>CUDA_LAUNCH_BLOCKING=1 python script.py args</code> and post the stack trace here?<br>
Also, is your code working fine on the CPU?</p>

username: chaslie
cooked: <p>Hi ptrblck,</p>
<p>How are you?</p>
<p>I am curretly running the model with:</p>
<pre><code class="lang-auto">import os
os.environ['CUDA_LAUNCH_BLOCKING'] = "1"
</code></pre>
<p>I haven’t tried to run it on the CPU. I will copy the stack trace whenn it fails at epoch 39.</p>
<p>this is releated to my other post about NaN when using <strong>torch.nn.BCEWithLogitsLoss</strong> at epoch 39…</p>
<p>cheers for now,</p>
<p>chaslie</p>

username: chaslie
cooked: <p>hi Ptrblck,</p>
<p>The model has completed its run with</p>
<pre><code class="lang-auto">## debugging the remove when debugged
import os
os.environ['CUDA_LAUNCH_BLOCKING'] = "1"
</code></pre>
<p>and the result is:</p>
<pre><code class="lang-auto">31 6912 tensor(298.7733, device='cuda:0', grad_fn=&lt;NegBackward&gt;)
32 6912 tensor(287.6319, device='cuda:0', grad_fn=&lt;NegBackward&gt;)
33 6912 tensor(345.7334, device='cuda:0', grad_fn=&lt;NegBackward&gt;)
34 6912 tensor(299.0026, device='cuda:0', grad_fn=&lt;NegBackward&gt;)
35 6912 tensor(336.5276, device='cuda:0', grad_fn=&lt;NegBackward&gt;)
36 6912 tensor(304.2394, device='cuda:0', grad_fn=&lt;NegBackward&gt;)
37 6912 tensor(274.8873, device='cuda:0', grad_fn=&lt;NegBackward&gt;)
38 6912 tensor(328.6809, device='cuda:0', grad_fn=&lt;NegBackward&gt;)
Traceback (most recent call last):

    logvar,mean,loss,out,logqz_x,logpz,logpx_z,z2 = loss_fn(model, data)

 XXXXXXX line 286, in loss_fn
    logpx_z=-torch.sum(BCE,[1,2],keepdim=False)

RuntimeError: CUDA error: device-side assert triggered
</code></pre>
<p>the function with the code of question is:</p>
<pre><code class="lang-auto">def loss_fn(model, data):
    mean, logvar = model.encode(data)
    z2=model.reparm(mean,logvar)
    out=model.decode(z2)

    criterion = torch.nn.BCELoss(size_average=False,reduce=False, reduction='sum')
    #criterion = torch.nn.BCEWithLogitsLoss(size_average=False,reduce=False, reduction='sum')
    BCE=criterion(out,data)
    logpx_z=-torch.sum(BCE,[1,2],keepdim=False)
    #logpx_z=-torch.sum(BCE,2,keepdim=True)
    logpz=log_normal_pdf(z2,torch.tensor(0.),torch.tensor(0.))
    logqz_x=log_normal_pdf(z2, mean, logvar)
    mean=logpx_z+logpz-logqz_x
    
    loss=-torch.mean(mean)
    return logvar,mean,loss,out,logqz_x,logpz,logpx_z,z2
</code></pre>
<p>I really have no idea why this should be failing because when i look at the output for logpz in the past there doesn’t seem to be anything strange.</p>
<p>The model is running on the cpu:</p>
<pre><code class="lang-auto">  warnings.warn(warning.format(ret))
1 6912 tensor(627.7763, grad_fn=&lt;NegBackward&gt;)
</code></pre>
<p>regards,</p>
<p>chaslie</p>

username: ptrblck
cooked: <p>Thanks for the debugging!</p>
<p>Could you post the stats of <code>BCE</code>, before this assertion is triggered?<br>
Using the stats of the tensor (min/max/mean), we could try to rerun the offending line of code with random values until we run into the same error and can debug further.</p>

username: chaslie
cooked: <p>hi ptrblck,</p>
<p>how do i do that?</p>
<p>the model has finished running on the cpu with the following:</p>
<pre><code class="lang-auto">37 6912 tensor(284.1459, grad_fn=&lt;NegBackward&gt;)
38 6912 tensor(298.0830, grad_fn=&lt;NegBackward&gt;)
Traceback (most recent call last):

logvar,mean,loss,out,logqz_x,logpz,logpx_z,z2 = loss_fn(model, data)
BCE=criterion(out,data)
result = self.forward(*input, **kwargs)
return F.binary_cross_entropy(input, target, weight=self.weight, reduction=self.reduction)

 \site-packages\torch\nn\functional.py", line 2077, in binary_cross_entropy
    input, target, weight, reduction_enum)

RuntimeError: Assertion `x &gt;= 0. &amp;&amp; x &lt;= 1.' failed. input value should be between 0~1, but got -nan(ind) at C:\w\1\s\tmp_conda_3.7_100118\conda\conda-bld\pytorch_1579082551706\work\aten\src\THNN/generic/BCECriterion.c:34
</code></pre>
<p>chaslie</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="chaslie" data-post="6" data-topic="74939">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/c/9fc29f/48.png" class="avatar"> chaslie:</div>
<blockquote>
<p>how do i do that?</p>
</blockquote>
</aside>
<p>Add the print statements before the loss calculation via:</p>
<pre data-code-wrap="python"><code class="lang-python">print(input.mean(), input.min(). input.max())
print(target.mean(), ...)
</code></pre>
<p>We could use these stats to create random tensors and try to run into this issue locally.</p>
<p>Also, could you check your numpy version via <code>print(np.__version__)</code>, please?</p>

username: chaslie
cooked: <p>hi ptrblck,</p>
<p>will do but it will take a couple of hours to run…</p>
<p>numpy version is 1.18.1</p>
<p>chaslie</p>

username: chaslie
cooked: <p>Hi Ptrblck,</p>
<p>I have made a couple of changes to the code before rerunning. I have changed the logpx_z function to:</p>
<p><code>logpx_z=-torch.sum(BCE,[0,1,2],keepdim=False)</code></p>
<p>The model has now completed epoch 40:</p>
<pre><code class="lang-auto">40 6912 tensor(10543.6299, device='cuda:0', grad_fn=&lt;NegBackward&gt;) tensor(0.2380, device='cuda:0') tensor(3.9410e-06, device='cuda:0') tensor(1., device='cuda:0') tensor(0.2542, device='cuda:0', grad_fn=&lt;MeanBackward0&gt;) tensor(0.0054, device='cuda:0', grad_fn=&lt;MinBackward1&gt;) tensor(1., device='cuda:0', grad_fn=&lt;MaxBackward1&gt;)
</code></pre>
<p>the Cuda Launch Blocking yields:</p>
<pre><code class="lang-auto">line 334, in &lt;module&gt;
loss.backward()

\torch\tensor.py", line 195, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph)

\torch\autograd\__init__.py", line 99, in backward
allow_unreachable=True)  # allow_unreachable flag
</code></pre>
<p>I also have the total output of BCE for epoch 37,38,39 &amp; 40 dumped into a pickle file.</p>
<p>chaslie</p>

username: ptrblck
cooked: <p>Could you upload these files?<br>
Also, which PyTorch version are you using?</p>

username: chaslie
cooked: <p>Hi Ptrblck,</p>
<p>I am currently using pytorch 1.4.0</p>
<blockquote>
<p>import torch<br>
print(torch.<strong>version</strong>)<br>
1.4.0</p>
</blockquote>
<p>Unfortunataly the BCE dump is over 3GB and as a result is abit unweildy…</p>
<p>I am re-running asking for the min max and mean from the output for each epoch up to 38 and then at each increment from epoch 38 onwards (along with output from the decoder).</p>
<p>I will take a screen grab for the last few iteration before fail and copy them in to the post.</p>
<p>do you have any thoughts on what maybe causing this?</p>
<p>One thing though, it seems to be generating a -ve minimum BCE value:</p>
<blockquote>
<p>7 6912 tensor(13018.4346, device=‘cuda:0’, grad_fn=) tensor(0.5092, device=‘cuda:0’, grad_fn=) <strong>tensor(-0., device=‘cuda:0’, grad_fn=)</strong> tensor(1.5307, device=‘cuda:0’, grad_fn=) tensor(0.3879, device=‘cuda:0’, grad_fn=) tensor(0.1181, device=‘cuda:0’, grad_fn=) tensor(1., device=‘cuda:0’, grad_fn=)</p>
</blockquote>
<p>chaslie</p>

username: ptrblck
cooked: <p>Thanks for the update!<br>
Is the <code>-0</code> tensor the sigmoid output of your model?</p>

username: chaslie
cooked: <p>its the BCE output, I have used the following within the class model, (which i assume is applying a sigmoid function):</p>
<pre><code class="lang-auto">    def decode(self, z2,apply_sigmoid=True):
        logits=self.Decoder(z2)
        if apply_sigmoid:
            probs = torch.sigmoid(logits)
            return probs
    
        return logits
</code></pre>
<p>“z2” is the input into the decoder, which returns “out” and “out” is compared against “data” in BCE function…</p>
<p>I have tried to utilise the BCEwithLogits as well as the above function and this also failed at epoch 38ish…</p>
<p>There is something strange happening in that region but i have looked at all the outputs for epoch 36 onwards and cannot see the problem, i am begining to think that the problem is more fundamental…</p>