TypeError: can't convert np.ndarray of type numpy.object_

username: Deb_Prakash_Chatterj
cooked: <p>This is my data,</p>
<div class="md-table">
<table>
<thead>
<tr>
<th></th>
<th>id</th>
<th>label</th>
<th>tweet</th>
</tr>
</thead>
<tbody>
<tr>
<td>0</td>
<td>1</td>
<td>0</td>
<td>
<span class="mention">@user</span> when a father is dysfunctional and is so selfish he drags his kids into his dysfunction. <span class="hashtag">#run</span>
</td>
</tr>
</tbody>
</table>
</div><p>which is in text format, I have pre-processed it and then I want to fit a PyTorch LSTM model in it.<br>
To fit the model I have to split the dataset into train and test set, and as PyTorch has a very interesting module called DataLoader to load the dataset, so we could use it. But as soon as I do this -</p>
<pre><code class="lang-auto">train_data = TensorDataset(torch.from_numpy(np.array(train_x)), torch.from_numpy(np.array(train_y)))
valid_data = TensorDataset(torch.from_numpy(np.array(valid_x)), torch.from_numpy(np.array(valid_y)))
test_data = TensorDataset(torch.from_numpy(np.array(test_x)), torch.from_numpy(np.array(test_y)))
</code></pre>
<p>It throws an error that<br>
<code>TypeError: can't convert np.ndarray of type numpy.object_. The only supported types are: float64, float32, float16, int64, int32, int16, int8, and uint8.</code> in the line of</p>
<pre><code class="lang-auto">----&gt; 4 train_data = TensorDataset(torch.from_numpy(np.array(train_x)),torch.from_numpy(np.array(train_y)))
</code></pre>
<p>I have also printed the shape and type of the splitted datasets,</p>
<pre><code class="lang-auto">print('shape of training set: {}' .format(train_x.shape))
print('shape of valid set: {}' .format(valid_x.shape))
print('shape of test set: {}' .format(test_x.shape))

shape of training set: (32979,)
shape of valid set: (2910,)
shape of test set: (2910,)
</code></pre>
<pre><code class="lang-auto">print(train_x.dtype)

object
</code></pre>
<p>How can I solve this error? I have tried solutions like converting the object type to string or float, neither of them worked. I am not getting any solutions. Any help appriciated.</p>

username: ptrblck
cooked: <p>I would suggest to encode the words as some indices similar to the <code>Lang</code> class in the <a href="https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html" rel="nofollow noopener">Seq2Seq Tutorial</a>.</p>

username: Deb_Prakash_Chatterj
cooked: <p>So, here I am encoding my sorted words using this -</p>
<pre><code class="lang-auto">reviews_int = []
for review in combi['tidy_tweet']:
  r = [vocab_to_int[w] for w in review.split()]
  reviews_int.append(r)
print(reviews_int[0:3])
OUTPUT: [[38, 4, 86, 10, 14869, 6, 10, 22, 4161, 71, 9356, 97, 332, 245, 97, 14870, 1301], [170, 8, 11409, 2355, 3, 33, 18, 424, 625, 59, 70, 18, 1651, 14871, 14872, 7, 14873, 22887, 14874], [76, 25, 4162]]
</code></pre>
<p>Then I convert it to np.array like this - <code>reviews_int = np.array(reviews_int)</code><br>
Still when I try to print the dtype of the <code>reviews_int</code>, it shows -<br>
<code>print(reviews_int.dtype) object</code><br>
How can I do this now? Is Seq2Seq is the only way? My approach is wrong?</p>

username: ptrblck
cooked: <p>I’m not sure, how you’ve processed the text, but the methods from the seq2seq tutorial seem to work for your example tweet:</p>
<pre><code class="lang-python">text = '@user when a father is dysfunctional and is so selfish he drags his kids into his dysfunction. #run'
text = normalizeString(text)
input_lang = Lang('tweet')
input_lang.addSentence(text)

encoded = [input_lang.word2index[w] for w in text.split(' ')]
input = torch.tensor(encoded)
print(input.type())
&gt; torch.LongTensor
</code></pre>

username: Deb_Prakash_Chatterj
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="4" data-topic="50014">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>normalizeString</p>
</blockquote>
</aside>
<p>So, do I need apply those two methods?<br>
<code>normalizeString</code> and <code>Lang</code>?<br>
Also, what does it <code>input_lang = Lang('tweet')</code> mean?<br>
why aren’t we passing <code>text</code> file there?</p>

username: ptrblck
cooked: <p>You don’t need the exact same class <code>Lang</code>, but could have a look at the underlying operations and how to transform each word to an index.<br>
Also, zou don’t necessarily need <code>normalizeString</code>, but it might help cleaning the tweets.</p>
<p>The passed argument to <code>Lang</code> is just the name of the language, which I called “tweet”.</p>
<p>The most important part is, that using the <code>input_lang.word2index</code> dict, you’ll get valid word indices, which can be used to train a model.</p>