GPU integer multiplication overflow

bogdan_ka: <p>Hi I have a simple model below that runs on cpu and returns the output as expected. The moment I set this model to CUDA I get the interger multiplication overflow issue. This model only embeds a set of integers so it’s not clear to me where this is coming from.</p>
<pre><code class="lang-auto">import torch
import numpy

device = (torch.device('cuda') if torch.cuda.is_available()
else torch.device('cpu'))

class SmallModel(torch.nn.Module):
    def __init__(self, skill_num, emb_size, max_seq_length):
        super().__init__()
       
        self.emb_size=emb_size
        self.skill_num=skill_num
        self.max_seq_length=max_seq_length
       
        self.skill_embeddings=torch.nn.Embedding(self.skill_num, self.emb_size)
        self.inter_embeddings=torch.nn.Embedding(self.skill_num*2,self.emb_size)
        self.embd_pos = torch.nn.Embedding(self.max_seq_length , self.emb_size)

         
    def forward(self, x, y):
        query = self.skill_embeddings(x) # shape bs X seq_len X emb_size 
#         mask_labels = y * (y &gt; -1).long()
#         #mask_labels=mask_labels.to(device)
#         key = self.inter_embeddings(x+mask_labels*self.skill_num)
#         values = self.inter_embeddings(x+mask_labels*self.skill_num)
#         pos = self.embd_pos(torch.arange(x.shape[1]))
#         key = key+pos 
#         query = query+pos 
        
        return query

# create some mock data, 5 students with 10 seq length
input=abs(torch.ceil(torch.randn(5,10)*100)).type(torch.int)
output=torch.zeros(5,10).type(torch.int) 
output[1,3]=1
output[2,5]=1
output[4,9]=-1

# set model params
skill_num=int(torch.max(input).numpy())+1
emb_size=12
max_seq_length=10

# init model
test_mod=SmallModel(skill_num=skill_num, emb_size=emb_size, max_seq_length=max_seq_length).to(device)

# run 
#query=test_mod(input, output)  # with cpu
query=test_mod(input.to(device), output.to(device)) # with gpu
query
</code></pre>
<p>Error message</p>
<pre><code class="lang-auto">RuntimeError                              Traceback (most recent call last)
/opt/conda/lib/python3.7/site-packages/IPython/core/formatters.py in __call__(self, obj)
    700                 type_pprinters=self.type_printers,
    701                 deferred_pprinters=self.deferred_printers)
--&gt; 702             printer.pretty(obj)
    703             printer.flush()
    704             return stream.getvalue()

/opt/conda/lib/python3.7/site-packages/IPython/lib/pretty.py in pretty(self, obj)
    392                         if cls is not object \
    393                                 and callable(cls.__dict__.get('__repr__')):
--&gt; 394                             return _repr_pprint(obj, self, cycle)
    395 
    396             return _default_pprint(obj, self, cycle)

/opt/conda/lib/python3.7/site-packages/IPython/lib/pretty.py in _repr_pprint(obj, p, cycle)
    698     """A pprint that just redirects to the normal repr function."""
    699     # Find newlines and replace them with p.break_()
--&gt; 700     output = repr(obj)
    701     lines = output.splitlines()
    702     with p.group():

/opt/conda/lib/python3.7/site-packages/torch/_tensor.py in __repr__(self, tensor_contents)
    425             )
    426         # All strings are unicode in Python 3.
--&gt; 427         return torch._tensor_str._str(self, tensor_contents=tensor_contents)
    428 
    429     def backward(

/opt/conda/lib/python3.7/site-packages/torch/_tensor_str.py in _str(self, tensor_contents)
    635     with torch.no_grad():
    636         guard = torch._C._DisableFuncTorch()
--&gt; 637         return _str_intern(self, tensor_contents=tensor_contents)

/opt/conda/lib/python3.7/site-packages/torch/_tensor_str.py in _str_intern(inp, tensor_contents)
    566                         tensor_str = _tensor_str(self.to_dense(), indent)
    567                     else:
--&gt; 568                         tensor_str = _tensor_str(self, indent)
    569 
    570     if self.layout != torch.strided:

/opt/conda/lib/python3.7/site-packages/torch/_tensor_str.py in _tensor_str(self, indent)
    326         )
    327     else:
--&gt; 328         formatter = _Formatter(get_summarized_data(self) if summarize else self)
    329         return _tensor_str_with_formatter(self, indent, summarize, formatter)
    330 

/opt/conda/lib/python3.7/site-packages/torch/_tensor_str.py in __init__(self, tensor)
    114         else:
    115             nonzero_finite_vals = torch.masked_select(
--&gt; 116                 tensor_view, torch.isfinite(tensor_view) &amp; tensor_view.ne(0)
    117             )
    118 

RuntimeError: numel: integer multiplication overflow


</code></pre>

ptrblck: <p>Could you check the number of elements in your input and if this would overflow <code>int32</code>?<br>
If so, you might need to use <code>long</code> as the <code>dtype</code>.</p>

bogdan_ka: <p>Thank you for the response.</p>
<p><code>torch.numel(input)</code> gives 50, which I think is what I expect by setting dim to (5,10). I tried converting input into long tensor as below. Still get the same error.</p>
<p><code>input=input.type(torch.LongTensor)</code></p>
<p>And to be clear this only happens on cuda, works fine on cpu.</p>

ptrblck: <p>I cannot reproduce the issue using your code on my GPU with <code>2.1.0.dev20230605+cu121</code> and based on the provided shapes the error should never be raised.</p>

bogdan_ka: <p>Good to know. I wonder if it’s the version issue.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/d/9df4f685f434bf2f758b7542f9ff2fb67ed1987d.png" alt="image" data-base62-sha1="mxlNcHJhX7xiBMgv3QabvcpWhQx" width="290" height="79"></p>

ptrblck: <p>I doubt it, as I also cannot reproduce it in <code>1.13.1+cu117</code>.</p>

bogdan_ka: <p>Ok this is an extremely bizzare thing. I reduced it all to bare bones, and made the input completely explicit. The embedding layer outside of the model returns what it should. The model itself runs and if I call <code>query.shape</code> I get the right shape output. The moment I call <code>query</code> to get the output I get the same int overflow error. This is on Vertex AI, python version 3.7.12.</p>
<p>Is this possibly because output  (query) is on gpu and calling it results in an error? I don’t have a clue what else could be going on on something this simple.</p>
<pre><code class="lang-auto">import torch
import numpy

device='cuda'

class SmallModel(torch.nn.Module):
    def __init__(self, skill_num, emb_size):
        super().__init__()
       
        self.emb_size=emb_size
        self.skill_num=skill_num
        self.skill_embeddings=torch.nn.Embedding(self.skill_num, self.emb_size)
                 
    def forward(self, x):
        query = self.skill_embeddings(x) 
        
        return query

inp=torch.tensor([[1,2,3,4],[4,5,2,3]])
test_mod=SmallModel(skill_num=6, emb_size=12).to(device)

out=test_mod(inp.to(device)) 
out.shape
</code></pre>

bogdan_ka: <p>Follow up<br>
<code>out.to('cpu')</code> gets rid of the error</p>

ptrblck: <p>I don’t know what Vertex AI is, but I’m just printing the <code>CUDATensor</code> and get the valid result.<br>
For the sake of completeness: the error is raised <a href="https://github.com/pytorch/pytorch/blob/f7fdaf8191ded1975d6119c8efe4a75953ca720e/c10/core/TensorImpl.h#L2493-L2505">here</a> and I don’t know how it can fail in your setup.</p>

bogdan_ka: <p>Thanks for the help. Vertex AI is just google’s cloud computing platfrom. To close this out, I restarted the kernel and added this line before setting the device and the problem seems to be gone.</p>
<pre><code class="lang-auto">
os.environ['CUDA_LAUNCH_BLOCKING'] = "1"
</code></pre>