Reduce failed to synchronize?

ZeweiChu: <p>I got this error when I compute the BCELoss, any idea what might be the reason?</p>
<blockquote>
<p>Traceback (most recent call last):<br>
File “main.py”, line 255, in <br>
train(args)<br>
File “main.py”, line 204, in train<br>
train_loss, train_acc = train_epoch(train_dataloader, model, crit, optimizer, args, reverse_dictionary)<br>
File “main.py”, line 67, in train_epoch<br>
loss = crit(pred, label)<br>
File “/share/data/speech/zewei/anaconda3/envs/py36/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 325, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/share/data/speech/zewei/anaconda3/envs/py36/lib/python3.6/site-packages/torch/nn/modules/loss.py”, line 372, in forward<br>
size_average=self.size_average)<br>
File “/share/data/speech/zewei/anaconda3/envs/py36/lib/python3.6/site-packages/torch/nn/functional.py”, line 1179, in binary_cross_entropy<br>
return torch._C._nn.binary_cross_entropy(input, target, weight, size_average)<br>
RuntimeError: reduce failed to synchronize: device-side assert triggered</p>
</blockquote>
<p>an it comes with 128 assertion fails like the following:</p>
<blockquote>
<p>/pytorch/torch/lib/THCUNN/BCECriterion.cu:30: Acctype bce_functor&lt;Dtype, Acctype&gt;::operator()(Tuple) [with Tuple = thrust::detail::tuple_of_iterator_references&lt;thrust::device_reference, thrust::device_reference, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type&gt;, Dtype = float, Acctype = float]: block: [0,0,0], thread: [125,0,0] Assertion <code>input &gt;= 0. &amp;&amp; input &lt;= 1.</code> failed.<br>
/pytorch/torch/lib/THCUNN/BCECriterion.cu:30: Acctype bce_functor&lt;Dtype, Acctype&gt;::operator()(Tuple) [with Tuple = thrust::detail::tuple_of_iterator_references&lt;thrust::device_reference, thrust::device_reference, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type&gt;, Dtype = float, Acctype = float]: block: [0,0,0], thread: [126,0,0] Assertion <code>input &gt;= 0. &amp;&amp; input &lt;= 1.</code> failed.<br>
/pytorch/torch/lib/THCUNN/BCECriterion.cu:30: Acctype bce_functor&lt;Dtype, Acctype&gt;::operator()(Tuple) [with Tuple = thrust::detail::tuple_of_iterator_references&lt;thrust::device_reference, thrust::device_reference, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type, thrust::null_type&gt;, Dtype = float, Acctype = float]: block: [0,0,0], thread: [127,0,0] Assertion <code>input &gt;= 0. &amp;&amp; input &lt;= 1.</code> failed.</p>
</blockquote>

richard: <p>Could you check that your <code>input</code> tensor has values that are between 0 and 1?<br>
You could add the following line to right before your loss call:<br>
<code>assert (input &gt;= 0. &amp; input &lt;= 1.).all()</code><br>
and see if it’s ever triggered.</p>

Naveed_Riaz: <p><a class="mention" href="/u/richard">@richard</a> i have tried you solution but it did not work for me and i got an error like<br>
assert (input &gt;= 0. &amp; input &lt;= 1.).all()<br>
TypeError: unsupported operand type(s) for &amp;: ‘float’ and ‘builtin_function_or_method’</p>

Xiaoyu_Song: <p>Hi, I tried your solution but I get an error below:</p>
<pre><code class="lang-auto">TypeError: unsupported operand type(s) for &amp;: 'float' and 'Tensor'
</code></pre>
<p>could you give me more advice?</p>

Naveed_Riaz: <p>In my case, this solution did not work. So i just debug my code line by line and got some value errors. So first debug your code is better solution. This solution is effective some specific cases.</p>

churchillmic: <p>I got this error, but it was because my data had an Inf in it (between 0 and 1 is not a true error, this data usually higher than 1 and does fine).</p>

Ray_Luo: <p>do <code>assert (x.data.cpu().numpy().all() &gt;= 0. and x.data.cpu().numpy().all() &lt;= 1.)</code></p>

Koke_Cacao: <p>I had the problem too, but the inspirations above helped me.<br>
In the end, I forgot to pass the Sigmoid function (or whatever function you use to get the probability)</p>
<pre><code class="lang-auto">prob_predict = torch.nn.Sigmoid()(logits_predict)
</code></pre>

xuebofan: <p>Assuming you have a N * M tensor, you can use the following code as a sanity check</p>
<p>assert(all([val &gt;= 0 and val &lt;= 1 for row in x.cpu().detach().numpy() for val in row]))</p>
<p>If this assertion fail, then you just need to normalize the output by adding a Sigmoid to x to resolve the problem. Otherwise there’s some other problems</p>