Torch.tensor() throws "RuntimeError: Overflow when unpacking long" exception?!

rbelew: <p>i see this exception mentioned in some older and more exotic contexts but this is as straight-forward as it gets:</p>
<pre><code class="lang-auto">    import torch

    for package in [torch]: 
        print(package.__name__, package.__version__)
    
    tstList = [10,20,30,40,50]
    tstTensor = torch.tensor(tstList)
    print('tst',type(tstTensor),tstTensor)
    
    tstLongTensor = torch.tensor(tstList,dtype=torch.int64)
    print('long',type(tstLongTensor),tstLongTensor)
    
    tstLng2Tensor = torch.tensor(tstList).type(torch.int64)
    print('long2',type(tstLng2Tensor),tstLng2Tensor) 

    indexedTst = [10239237003504588839, 9921686513378912864, 11901859001352538922, 4316640507316845735, 7162342725099726394, 17494803046312582752]
    idxTensor = torch.tensor(indexedTst)
    print('idx',type(idxTensor),idxTensor)
</code></pre>
<p>works fine until it’s given long ints:</p>
<pre><code class="lang-auto">torch 2.3.0
tst &lt;class 'torch.Tensor'&gt; tensor([10, 20, 30, 40, 50])
long &lt;class 'torch.Tensor'&gt; tensor([10, 20, 30, 40, 50])
long2 &lt;class 'torch.Tensor'&gt; tensor([10, 20, 30, 40, 50])
Traceback (most recent call last):
  File ".../tstTensor.py", line 33, in &lt;module&gt;
    main()
  File ".../tstTensor.py", line 29, in main
    idxTensor = torch.tensor(indexedTst)
                ^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Overflow when unpacking long
</code></pre>
<p>version info:</p>
<ul>
<li>Python 3.11.9 (main, Apr 19 2024, 11:43:47) [Clang 14.0.6 ] on darwin</li>
<li>same exception using device=<code>cpu</code> or <code>mps</code></li>
</ul>

ptrblck: <p>This error is expected since <code>torch.long==torch.int64</code> corresponds to a max. value of <code>9223372036854775807</code> while your values are larger than this:</p>
<pre data-code-wrap="python"><code class="lang-python">torch.iinfo(torch.long).max &gt; 10239237003504588839
# False
</code></pre>
<p>Use <code>torch.uint64</code> if you really need such large values:</p>
<pre data-code-wrap="python"><code class="lang-python">indexedTst = [10239237003504588839, 9921686513378912864, 11901859001352538922, 4316640507316845735, 7162342725099726394, 17494803046312582752]
idxTensor = torch.tensor(indexedTst, dtype=torch.uint64)
print('idx',type(idxTensor),idxTensor)
# idx &lt;class 'torch.Tensor'&gt; tensor([10239237003504588839,  9921686513378912864, 11901859001352538922,
#          4316640507316845735,  7162342725099726394, 17494803046312582752],
#        dtype=torch.uint64)
</code></pre>
<p>I would also be interested to learn more about your use case and when these indices are useful.</p>

rbelew: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="209386">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p><code>idxTensor = torch.tensor(indexedTst, dtype=torch.uint64)</code></p>
</blockquote>
</aside>
<p>Excellent, thanks so much <a class="mention" href="/u/ptrblck">@ptrblck</a> ! i had been looking to find the max value, now i know about <code>torch.iinfor()</code>!</p>
<blockquote>
<p>I would also be interested to learn more about your use case and when these indices are useful.</p>
</blockquote>
<p>I am getting these large ints from <a href="https://spacy.io/" rel="noopener nofollow ugc">spaCy</a>’s token hashing function. Spacy uses hashing on texts to get unique ids (cf. <a href="https://stackoverflow.com/a/53213295/1079688" rel="noopener nofollow ugc">SO</a> )</p>
<pre><code class="lang-auto">&gt;&gt;&gt; import spacy
&gt;&gt;&gt; nlp = spacy.load('en')
&gt;&gt;&gt; text = "here is some test text"
&gt;&gt;&gt; doc = nlp(text)
&gt;&gt;&gt; [token.norm for token in doc]
[411390626470654571, 3411606890003347522, 7000492816108906599, 1618900948208871284, 15099781594404091470]
</code></pre>

ptrblck: <p>Ah, this is indeed an interesting use case! Thanks for sharing as I was wondering how large your tensors are to need such large indexing values <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"></p>

rbelew: <p>hmm, but now <code>embedding()</code> doesn’t know how to deal with these big uint64 <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=12" title=":frowning:" class="emoji" alt=":frowning:" loading="lazy" width="20" height="20"></p>
<p><code>RuntimeError: Expected tensor for argument #1 'indices' to have one of the following scalar types: Long, Int; but got MPSUInt64Type instead (while checking arguments for embedding)</code></p>

ptrblck: <p>OK, in this case I might not fully understand your use case yet.<br>
An <code>nn.Embedding</code> layer is creating an internal <code>weight</code> matrix (in <code>float32</code> by default) in the shape <code>[num_embeddings, embedding_dim]</code>, which is indexed with the input tensor.<br>
Here is a small example:</p>
<pre data-code-wrap="python"><code class="lang-python">num_embeddings = 10
embedding_dim = 100
emb = nn.Embedding(num_embeddings, embedding_dim)

print(emb.weight.shape)
# torch.Size([10, 100])

x = torch.tensor([0, 4, 9])
out = emb(x)
print(out.shape)
# torch.Size([3, 100])

print((out[0] == emb.weight[0]).all())
# tensor(True)
print((out[1] == emb.weight[4]).all())
# tensor(True)
print((out[2] == emb.weight[9]).all())
# tensor(True)
</code></pre>
<p>Note that the input indices should have the range <code>[0, num_embeddings-1]</code>. Out-of-range values will trigger an error:</p>
<pre data-code-wrap="python"><code class="lang-python">x = torch.tensor([10])
out = emb(x)
# IndexError: index out of range in self
</code></pre>
<p>Your input seems to be a hash value with a max. value of <code>&gt;=10239237003504588839</code>.<br>
If you would thus try to create a <code>weight</code> matrix with <code>num_embeddings=10239237003504588839</code> and even with <code>embedding_dim=1</code>, the <code>weight</code> matrix would use <code>10239237003504588839 * 4 / 1024**3 = 38144130272.81 GB</code> of memory. Even if you could use 1-bit to store the embedding <code>weight</code> it would still use <code>10239237003504588839 / 1024**3 = 9536032568.20 GB</code>.</p>
<p>Could you double check if you really want to use these has values as indices or would rather want to remap them to <code>[0, num_embeddings-1]</code>?</p>

rbelew: <p>you make very good sense.  the indices were intended for use with a <a href="https://captum.ai/tutorials/IMDB_TorchText_Interpret" rel="noopener nofollow ugc">captum tutorial</a> that depended on a pre-trained CNN model, but you help me see that trying to build my own spacy token-&gt;index vocabulary isn’t the way.  i created a different post: <a href="https://discuss.pytorch.org/t/imdb-torchtext-interpret-tutorial-uses-deprecated-torchtext/209536" class="inline-onebox">IMDB_TorchText_Interpret tutorial uses deprecated torchtext</a>.</p>