Simple code runs on GPU but CPU

thiennp: <p>I’m not sure this is a bug or not.</p>
<p>I need to deploy the AWS elastic inference for our service. The Elastic Inference requires using CPU to load and run models.</p>
<p>but our code runs well on GPUs, but CPU.</p>
<p>as the simple code below, it can be run on GPUs but on CPU it returns “index out of range in self” Error.</p>
<blockquote>
<h3>CPUs returns index out of range in self error</h3>
<p>import numpy as np<br>
import torch<br>
import torch.nn as nn</p>
<p>sinusoid_table = torch.FloatTensor(torch.Size([50 + 1, 512]))</p>
<p>pos_emb = nn.Embedding.from_pretrained(sinusoid_table, freeze=True)<br>
positions = torch.arange(200).expand(1, 200).contiguous()+1<br>
positions=positions<br>
a= pos_emb(positions)<br>
print(a)</p>
<h3>on GPUs this run well</h3>
<p>import torch<br>
import torch.nn as nn</p>
<p>device = torch.device(‘cuda:0’)</p>
<p>sinusoid_table = torch.FloatTensor(torch.Size([50 + 1, 512])).to(device)<br>
pos_emb = nn.Embedding.from_pretrained(sinusoid_table, freeze=True).to(device)<br>
positions = torch.arange(200).expand(1, 200).contiguous()+1<br>
positions=positions.to(device)<br>
a= pos_emb(positions)<br>
print(a)</p>
</blockquote>
<p>I’m using PytorchECL 1.3.1 (support Elastic Inference) comes with Amazon AMI</p>
<p>Tested with normal pytorch==1.5.0</p>
<p>both of them result the “index out of range error”</p>
<p>Any helps is appreciated. thank you.</p>

ptrblck: <p>The code raises an error on the CPU as well as the GPU using the latest stable version (<code>1.7.0</code>).<br>
Note that <code>assert</code> statements in CUDA code were mostly disabled in <code>1.5.0</code> due to a bug, so that your code doesn’t raise the proper error if you are using the GPU.</p>
<p>The reason for the error is that your <code>weight</code> lookup table in the embedding layer contains feature vectors for 51 indices, so your inputs are limited to <code>[0, 50]</code>, while you are passing <code>positions</code> with values in the range <code>[1, 200]</code>.</p>
<p>Also, note that <code>torch.FloatTensor(torch.Size([51, 512]))</code> will create a tensor in the specified shape with uninitialized values, which might contain invalid values such as NaNs and Infs, so you should properly initialize <code>sinusoid_table</code> or use a tensor factory method to create it such as <code>torch.randn</code>.</p>

thiennp: <p>Thank your sir.</p>
<p>Do you mean, with pytorch 1.5.0, it does have error, just disabled?<br>
But our project is running well. It’s so strange.</p>
<p>I did initialized with random values, just for simplicity, I removed the initializing script</p>
<p>Anyway, thank for your helps. I will try to fix this.</p>

ptrblck: <aside class="quote no-group" data-username="thiennp" data-post="3" data-topic="101556">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/thiennp/48/30601_2.png" class="avatar"> thiennp:</div>
<blockquote>
<p>Do you mean, with pytorch 1.5.0, it does have error, just disabled?<br>
But our project is running well. It’s so strange.</p>
</blockquote>
</aside>
<p>Yes, device assert statements were unfortunately disabled by accident (this should be fixed in <code>1.5.1</code> and all newer versions).<br>
I would strongly recommend to update PyTorch and to make sure your code is running properly.<br>
Your current code might run into an illegal memory access without reporting it and might thus create major issues.</p>