How to convert array to tensor?

jaeyung1001: <p>my data is like below:</p>
<p>X_train =<br>
[1,0,0,0,0,0]<br>
[0,0,0,0,0,1]<br>
[0,1,0,0,0,0]<br>
…</p>
<p>and I want to convert it tensor:<br>
<code>x_train_tensor = Variable(torch.Tensor(X_train.values))</code></p>
<p>but there is error like this:</p>
<blockquote>
<p>TypeError: can’t convert np.ndarray of type numpy.object_. The only supported types are: double, float, float16, int64, int32, and uint8.</p>
</blockquote>
<p>how can i fix this error?</p>

lelouedec: <p>use</p>
<blockquote>
<p>torch.from_numpy()</p>
</blockquote>

jaeyung1001: <p>the X_train type is Series</p>
<p>and i use:<br>
<code>torch.from_numpy(pd.Series.as_matrix(X_train))</code></p>
<p>but I met same error</p>

lelouedec: <p>Try</p>
<blockquote>
<p>torch.from_numpy(X_train.values)</p>
</blockquote>

jaeyung1001: <p>I met same Error… haha</p>

lelouedec: <p>What do you want to do exactly, X_train.values is giving you a numpy array, so torch.from_numpy should return correctly a Tensor.</p>

jaeyung1001: <p>I just want to convert my dataframe to tensor</p>
<p>here is return of X_train.values:</p>
<pre><code class="lang-auto">array([list([0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 1, 0, 4, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 1, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 1, 1, 0, 0, 1, 0, 0, 1, 1, 1, 0, 1, 1]),
       list([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]),
       list([0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 1, 0, 10, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 1, 1]),
...
</code></pre>

jaeyung1001: <p>Oh… Sorry I suppose my X_train data format is wrong… I resolve it like below:</p>
<pre><code class="lang-auto">temp = []
for i in X_train:
    temp.append(i)
</code></pre>
<p>and use torch.from_numpy(temp) and it’s work</p>
<p>thank you:)</p>

erikxiong: <p>Yes, you should set the values a 2D list like</p>
<pre><code class="lang-auto">x = torch.Tensor(list(X_train.values))
</code></pre>

Beyazit_Bestami_Yuks: <p>Now I’ve this one <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=9" title=":frowning:" class="emoji" alt=":frowning:"></p>
<p>Expected object of scalar type Double but got scalar type Float for argument <span class="hashtag">#2</span> ‘mat2’ in call to _th_mm</p>

ptrblck: <p>Make sure to pass the input tensor in the same data type as the layer parameters.<br>
This error is often raised, if you’ve created the input tensor from numpy arrays, since numpy uses float64 as the default type, while PyTorch uses float32.</p>

kendreaditya: <p>I got the same error:</p>
<pre><code class="lang-auto">dataset = torch.from_numpy(np.load("dataset.npy", allow_pickle=True))

TypeError: can't convert np.ndarray of type numpy.object_. The only supported types are: float64, float32, float16, int64, int32, int16, int8, uint8, and bool.
</code></pre>
<p>I tried converting the ndarray into a python list like this, but got another error (I think because its a 2D array/list)</p>
<pre><code class="lang-auto">dataset = np.load("dataset.npy", allow_pickle=True)
dataset = torch.Tensor(list(dataset))
ValueError: expected sequence of length 64 at dim 2 (got 2)
</code></pre>
<p>a object in the dataset has the shape of [[64,64],2]</p>

ptrblck: <p><code>torch.from_numpy</code> expects a single numpy array in the data types mentioned in the error message.<br>
Your <code>dataset.npy</code> seems to contain an array with different objects in it.<br>
I’m not sure if they only have a different shape or also differ in other attributes, but you would have to make sure to create a single array out of your data before passing it to <code>torch.from_numpy</code>.</p>

kendreaditya: <p>I solved the problem in the way you suggested by iteration through the data in single arrays, and thus rebuilding the dataset. Is there any way to convert all arrays within ndarrays to tensors?</p>

ptrblck: <p>Good to hear you solved it!<br>
Had all arrays the same shape inside the <code>ndarray</code>? If so, you could make sure to create a single numpy array, which doesn’t store each element as an <code>object</code>.<br>
Variant shapes will create such an “object array”:</p>
<pre><code class="lang-python">x = np.array([[1., 2.], [3., 4.]])
print(x.dtype)
&gt; float64
y = np.array([[1., 2.], [3.]])
print(y.dtype)
&gt; object
</code></pre>

kendreaditya: <p>I did have varying shapes, but I solved the problem by converting both the model data, and the one hot vector to tensors individually, so my code looked like this:</p>
<pre><code class="lang-auto"># temp contains NumPy objects
dataset = []
for object in temp:
    dataset.append([torch.Tensor(torch.Tensor(object[0])), torch.Tensor(object[1])])
    # object[0] contains the data; object[1] contains the one-hot vector
</code></pre>
<p>Since the numpy array contains objects instead of float64, or any other primitive data types, is there a way in avoiding the creation of another array?</p>

ptrblck: <p>I don’t think there is a simple operation for it and your approach looks valid.<br>
Once nested tensors are ready, it might be a simple <code>from_numpy</code> <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

wahyubram82: <p>i’m sorry, i’m new in pytorch, try to learn to load the dataset. but have a trouble with it.<br>
i have numpy data in pandas (from speech), how to load it to pytorch?<br>
my data in pandas looks like this:</p>
<pre><code class="lang-auto">array([array([[-0.00027978],
       [-0.00027978],
       [-0.00027978],
       ...,
       [ 0.02842325],
       [ 0.02795309],
       [ 0.03359528]], dtype=float32),
       array([[ 0.00029535],
       [ 0.00029535],
       [ 0.00029535],
       ...,
       [ 0.00383177],
       [ 0.00245112],
       [-0.00314868]], dtype=float32)], dtype=object)
</code></pre>
<p>it’s a data series…</p>
<p>error when try to make it as tensor dataset, the error is:<br>
<code>train_set = TensorDataset(torch.from_numpy(np.array(train_pd.segmentasi.values)))</code></p>
<p><code>TypeError: can't convert np.ndarray of type numpy.object_. The only supported types are: float64, float32, float16, int64, int32, int16, int8, uint8, and bool.</code></p>
<p>i think similar with the other question, but when i try to follow, nothing works for me…<br>
already read many question that maybe related, got the clue like this:</p>
<p><code>np.array((train_pd.segmen.values).tolist())</code></p>
<p>so, i make:<br>
train_set = TensorDataset(torch.from_numpy(np.array(train_pd.segmentasi.values).tolist())))<br>
still fail, because the array type become an object again.</p>
<p>then,try to set the dtype with:</p>
<p><code>np.array((train_pd.segmen.values).tolist(),dtype=np.float32)</code></p>
<p>with command:<br>
train_set = TensorDataset(torch.from_numpy(np.array((train_pd.segmentasi.values).tolist(),dtype=np.float32)))</p>
<p>back error, but this time said:</p>
<p><code>ValueError: setting an array element with a sequence</code></p>
<p>please…trully need an advice,  the king of novice with numpy and torch.</p>
<p>about the data, i already try to plot it with matplotlib, and it show picture of waveform, so i think the data is fine:</p>
<pre><code class="lang-auto">#test the data
segment_data = train_pd['segment'][2]
print(len(segment_data))

# show segmentation
plt.plot(segment_data)
plt.show()
</code></pre>
<pre><code class="lang-auto">156500
</code></pre>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/8/1/81a0e270da6c0410d960e31fc6ace7d8f83571f4.png" alt="index" data-base62-sha1="iuKhO2ZzXFTO6p962JEBjLWdfxy" width="381" height="248"></p>

ptrblck: <p>The issue is that your numpy array has <code>dtype=object</code>, which might come from mixed dtypes or shapes, if I’m not mistaken.<br>
The output also looks as if you are working with nested arrays. Could you try to print the shapes of all “internal” arrays and try to create a single array via e.g. <code>np.stack</code>?<br>
Once you have a single array with a valid <code>dtype</code>, you could use <code>torch.from_numpy</code>.</p>

wahyubram82: <p>I think…it’s not possible, my data resulting with the same function, a fragmented speech sound…</p>
<p>so…it’s similar with list of list, but in numpy form.<br>
each item, is a numpy float32 type.</p>
<p>all of that process save to pandas dataset.<br>
the problem is, because the ‘main list’ form is numpy object.<br>
the dataframe for each row is like this:</p>
<pre><code class="lang-auto">file_wav               transcription              segmendata
path/file1.wav     'one day vacation'     array([[-0.00027978],
                                                 [-0.00027978],
                                                 [-0.00027978],
                                                 [ 0.03359528]], dtype=float32)

</code></pre>
<p>so if i just call it only for one row, like:</p>
<p><code>segment_data = train_pd['segmendata'][4]</code></p>
<p>the form is like in dataframe above, that’s why I can plot it with matplotlib easily.</p>
<p>but when i call all of it with:</p>
<p><code>a = train_pd['segmendata'].values</code></p>
<p>the top bracket or top list or top cover of numpy cover it with dtype=object.</p>
<p>if like you you said, because of the different form item in pandas, the only possible things is: there is an empty item.</p>
<p>well, i must figure it out how to cek it</p>