How to deal with high-dim sparse tensors in torch?

SakuraiJJ: <p>I am finding a way to treat high-dim sparse tensors.</p>
<p>I downloaded <a href="https://archive.ics.uci.edu/dataset/26/connect+4" rel="noopener nofollow ugc">this data</a> from the UCI dataset. It is a tabular dataset with a size of 67557 × 42. Each row can take a categorical quantity (o, ×, b).</p>
<pre><code class="lang-auto">import numpy as np
import torch
import pandas as pd
from ucimlrepo import fetch_ucirepo
dataset = fetch_ucirepo(id=26)

X = dataset.data.features
X
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/f/3f40d83b98eaae94df6c21aae79f67ed01abd241.png" data-download-href="https://discuss.pytorch.org/uploads/default/3f40d83b98eaae94df6c21aae79f67ed01abd241" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/3/f/3f40d83b98eaae94df6c21aae79f67ed01abd241.png" alt="image" data-base62-sha1="91z0Bd58GUy4TiqFprpj9GMq0vf" width="517" height="297" data-dominant-color="F3F3F3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">984×567 19.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I want to convert this data into a sparse tensor as follows. Firstly, I replace (o, ×, b) with integers (0,1,2).</p>
<pre><code class="lang-auto">X = X.astype('category')
cat_columns = X.select_dtypes(['category']).columns
X[cat_columns] = X[cat_columns].apply(lambda x: x.cat.codes)
X
</code></pre>
<p>Now, <code>X</code> is an integer matrix. Then, I am trying to convert it into 42-dim sparse tensor.<br>
Each row of the table can be regarded as a 42-dim coordinate where the tensor has 1 value. The tensor values in other indices are 0.</p>
<pre><code class="lang-auto">X_np = np.transpose(X.to_numpy())
(D, N) = np.shape(X_np)
X_np = X_np.astype(np.int32)
X_th = torch.from_numpy(X_np)
X_torch = torch.sparse_coo_tensor(X_th, np.ones(N) )
</code></pre>
<p>However, I got the following error:</p>
<blockquote>
<p>RuntimeError<br>
Traceback (most recent call last)<br>
 in <br>
----&gt; 1 X_torch = torch.sparse_coo_tensor(X_th, np.ones(N) )<br>
RuntimeError: numel: integer multiplication overflow</p>
</blockquote>
<p>When the number of features of table data is less, I can use <code>torch.sparse_coo_tensor</code> in this way. For example, just changing <code>fetch_ucirepo(id=26)</code> into <code>fetch_ucirepo(id=101)</code> (<a href="https://archive.ics.uci.edu/dataset/101/tic+tac+toe+endgame" rel="noopener nofollow ugc">TicTacToe dataset</a>), it works.</p>
<p>I am just confused because, essentially, the data quantity of a sparse tensor is not huge, even if it is highly dimensional. So, why does this method not work? What is the alternative to treating high-dimensional sparse tensors?</p>
<p><strong>Note</strong>: The use of the UCI dataset above is for illustrative purposes, not because I want to turn table data into a tensor. I am looking for a way to handle high-dimensional sparse tensors in Torch.</p>

HalveLuve1022: <p>Hi there. It’s been almost one year and now I run into similar problems. I would like to build a torch.sparse_coo_tensor using a dictionary with massive amount of entries, and it always throws the error of <code>numel: integer multiplication overflow</code>. Did you manage to solve the problem?</p>