Device side assert from embedding lookup raises on subsequent CUDA connected instruction

username: drevicko
cooked: <p>I appear to be getting a device side assert from an embedding lookup, but (with CUDA_LAUNCH_BLOCKING enabled) the exception is raised for the next instruction that touches CUDA (in my case, it’s a call to <code>torch.zeros(..., device='cuda')</code>), several python instructions later.</p>
<pre><code class="lang-auto">entailed_embeds_calculated = [self.relation_embedding(index_tensor) for index_tensor in entailed_pred_indices]
# here self.relation_embedding is an nn.Embedding, and entailed_pred_indices is a list of index tensors.
# I checked, and an element of an index_tensor is out of bounds.

entailed_embeds_calculated = self.aggregator(entailed_embeds_calculated, entailed_scores)
# entailed_scores is also a list of tensors

...

def aggregator(embedding_lists, weights=None):
    # a few asserts and other non-CUDA python code
    embed_dim = embedding_lists[0].shape[-1]
    zero_tensor = torch.zeros((embed_dim,), dtype=torch.float, device=embedding_lists[0].device)
    # At this point the "Device side assert" is raised.
</code></pre>
<p>A similar behaviour appears to be happening <a href="https://discuss.pytorch.org/t/device-side-assert-triggered-at-error/82488">here</a>.</p>
<p>The CUDA error looks like this:</p>
<pre><code class="lang-auto">/opt/conda/conda-bld/pytorch_1579022060824/work/aten/src/THC/THCTensorIndex.cu:307: void indexSelectSmallIndex(TensorInfo&lt;T, IndexType&gt;, TensorInfo&lt;T, IndexType&gt;, TensorInfo&lt;long, IndexType&gt;, int, int, IndexType, long) [with T = float, IndexType = unsigned int, DstDim = 2, SrcDim = 2, IdxDim = -2]: block: [0,0,0], thread: [11,0,0] Assertion `srcIndex &lt; srcSelectDimSize` failed.

  File ".../RelationScorers.py", line 1072, in aggregator
    zero_tensor = torch.zeros((embed_dim,), dtype=torch.float, device=embedding_lists[0].device)
RuntimeError: CUDA error: device-side assert triggered
...</code></pre>

username: ptrblck
cooked: <p>Are you setting the <code>CUDA_LAUNCH_BLOCKING=1</code> env variable in your terminal or are you trying to set it in your script/notebook?<br>
In the latter case, please use it as an external env variable, as setting it via <code>os</code> inside a script will fail if any libraries were imported and have already initialized CUDA.</p>

username: drevicko
cooked: <p>I set it before launching python (ie: in the env) and reported the setting in python just to be sure.</p>
<p>I do see a difference when it’s not set: the exception is raised a few instructions later. So it seems to be doing something. It consistently raised at the same point (with CUDA_LAUNCH_BLOCKING set) for several runs and with differing GPU loads.</p>
<p>I’m running pytorch 1.4.0 (old for compatibility with HPC infrastructure) and CUDA Version 10.2.89. These aren’t the most recent, but I put this here partly for others who may encounter this odd behaviour to help them decifer it <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: ptrblck
cooked: <p>Could you update to the latest stable release and recheck this behavior?<br>
In your old version, device assert might have been missing, which would explain this behavior.<br>
In particular, this was the case in <code>1.5.0</code>, as device assertions were globally disabled (and fixed in <code>1.5.1</code>).<br>
I don’t recall, if there were similar issues in <code>1.4.0</code>.</p>