RuntimeError when using BertModel from Huggingface

username: attardi
cooked: <p>I have run into this error when using the BertModel from Huggingface transformers, version 2.10.0 while it works with version 2.1.1 and on CPU:</p>
<p>/pytorch/aten/src/THC/THCTensorIndex.cu:361: void indexSelectLargeIndex(TensorInfo&lt;T, IndexType&gt;, TensorInfo&lt;T, IndexT<br>
ype&gt;, TensorInfo&lt;long, IndexType&gt;, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, Ds<br>
tDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [145,0,0], thread: [95,0,0] Assertion <code>srcIndex &lt; srcS electDimSize</code> failed.<br>
Traceback (most recent call last):<br>
File “run.py”, line 58, in <br>
cmd(args)<br>
File “/project/piqasso/tools/biaffine-parser/parser/cmds/train.py”, line 82, in <strong>call</strong><br>
self.train(train.loader)<br>
File “/project/piqasso/tools/biaffine-parser/parser/cmds/cmd.py”, line 83, in train<br>
arc_scores, rel_scores = self.model(words, feats)<br>
File “/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py”, line 532, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/project/piqasso/tools/biaffine-parser/parser/model.py”, line 90, in forward<br>
feat_embed = self.feat_embed(*feats)<br>
File “/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py”, line 532, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/project/piqasso/tools/biaffine-parser/parser/modules/bert.py”, line 43, in forward<br>
bert = bert[bert_mask].split(bert_lens[mask].tolist())<br>
RuntimeError: copy_if failed to synchronize: cudaErrorAssert: device-side assert triggered</p>
<p>Any hints on how to find the cause?</p>
<p>Than you.</p>

username: attardi
cooked: <p>I believe the problem was due to a sentence of length 383, which was split into 672 wordpieces.</p>
<p>I traced te error to this call, in<br>
lib64/python3.6/site-packages/torch/nn/modules/sparse.py</p>
<p>111 -&gt;	def forward(self, input):<br>
112 return F.embedding(<br>
113 input, self.weight, self.padding_idx, self.max_norm,<br>
114 self.norm_type, self.scale_grad_by_freq, self.sparse)</p>
<p>where input = arange(672)<br>
(Pdb)<br>
–Return–<br>
THCudaCheck FAIL file=/pytorch/aten/src/THC/THCCachingHostAllocator.cpp line=278 error=700 : an illegal memory access was encountered</p>
<p>So it likely hits the limit of 512 of BERT inputs and breaks the memory.<br>
I wonder why they don’t check this limit and avoid memory corruption.</p>

username: ptrblck
cooked: <p>Based on the error message, it seems that you are using the GPU:</p>
<pre><code class="lang-python">/pytorch/aten/src/THC/THCTensorIndex.cu:361: void indexSelectLargeIndex(TensorInfo&lt;T, IndexType&gt;, TensorInfo&lt;T, IndexT
ype&gt;, TensorInfo&lt;long, IndexType&gt;, int, int, IndexType, IndexType, long) [with T = float, IndexType = unsigned int, Ds
tDim = 2, SrcDim = 2, IdxDim = -2, IndexIsMajor = true]: block: [145,0,0], thread: [95,0,0] Assertion srcIndex &lt; srcS electDimSize failed.
</code></pre>
<p>Which PyTorch version are you using? Could you update to the nightly binaries and rerun the code, please?</p>

username: attardi
cooked: <p>I tried also with torch 1.5.<br>
But it looks like an issue with torch.nn.embedding(), which does not check for the limit in its input vector.</p>

username: ptrblck
cooked: <p>Did the nightly binaries yield another error message? Recently an issue was fixed, which prevented the debugging asserts to raise a proper error message and let the code ran into an illegal memory access.</p>

username: attardi
cooked: <p>I haven’t tried that. I added a check in my code before invoking method forward(), which is worthwhile doing anyhow.<br>
Thank you anyhow.</p>