Load dataframe in torch

Adithya_Samavedhi: <p>Hi guys,<br>
im kinda new to pytorch,<br>
so i have a dataframe where one column is the output, and the other column contains values which are  of type ndarray, how am i supposed to load it from my pandas dataframe into torch so i can use it in a neural network model or cnn? i have attatched a snapshot of my dataframe.<br>
Thanks in advance<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/6/684f878b93bf8b94a9e15229c80132df8dad32a2.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/684f878b93bf8b94a9e15229c80132df8dad32a2" title="49"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/6/684f878b93bf8b94a9e15229c80132df8dad32a2_2_690x325.jpeg" alt="49" data-base62-sha1="eSM2YEM9yPqyPRsYqx5jsls8eRQ" width="690" height="325" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/6/684f878b93bf8b94a9e15229c80132df8dad32a2_2_690x325.jpeg, https://discuss.pytorch.org/uploads/default/optimized/2X/6/684f878b93bf8b94a9e15229c80132df8dad32a2_2_1035x487.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/6/684f878b93bf8b94a9e15229c80132df8dad32a2.jpeg 2x" data-dominant-color="9BA086"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">49</span><span class="informations">1152×544 49.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

Pranavan_Theivendira: <p>Hi,<br>
Try this.</p>
<pre><code class="lang-auto">import torch

torch_tensor_output = torch.tensor(df['output'].values)
torch_tensor_vectors = torch.from_numpy(df['vector'].values)
</code></pre>
<p>Hope this would help you.</p>

yuanzhoulvpi: <p>We use the iterators=True in the read_csv() function of pandas to read the csv loop into memory in batches. (If you don’t use the iterators parameter, the csv file with a large amount of data will be read into the memory. The memory is definitely  Not enough) The code is as follows:</p>
<pre><code class="lang-auto"># -*- coding: utf-8 -*-
 
import csv
import pandas as pd
import numpy as np
import torch
import torch.utils.data as data
 
 
class FaceLandmarksDataset(data.Dataset):
	"""Face Landmarks dataset."""
	def __init__(self, csv_file):
		"""
        Args:
            csv_file (string): Path to the csv file with annotations.
            root_dir (string): Directory with all the images.
            transform (callable, optional): Optional transform to be applied
                on a sample.
        """
		self.landmarks_frame = pd.read_csv(csv_file, iterator=True)
	def __len__(self):
		#print len(self.landmarks_frame)
		#return len(self.landmarks_frame)
		return 1800000
	def __getitem__(self, idx):
		print idx
		landmarks = self.landmarks_frame.get_chunk(128).as_matrix().astype('float')
		# landmarks = self.landmarks_frame.ix[idx, 1:].as_matrix().astype('float')
 
		# 
		return landmarks
 
 
filename = '/media/czn/e04e3ecf-cf63-416c-afd7-6d737e09968a/zhongkeyuan/dataset/CSV/HGG_pandas.csv'
dataset = FaceLandmarksDataset(filename)
train_loader = torch.utils.data.DataLoader(dataset, batch_size=1, shuffle=True)
for data in train_loader:
	print data


</code></pre>

Adithya_Samavedhi: <p>i tried that, but the second command gives this error,<br>
torch_tensor_vectors = torch.from_numpy(df[‘vector’].values) gives error-<br>
can’t convert a given np.ndarray to a tensor - it has an invalid type. The only supported types are: double, float, int64, int32, and uint8.</p>

Adithya_Samavedhi: <p>how will the program know which column is my label and which column is my matrix?</p>

Pranavan_Theivendira: <p>Hi Adithya,</p>
<p>Could you please share your panda data frame here?</p>
<p>I can have a look.</p>

god_sp33d: <p>Are are any missing values, nan or strings in the data ? Make sure of that.</p>

jayurbain: <p>I was able to correct this error by using reset_index on the dataframe prior to creating the torch tensor.</p>