How to load data from a .csv

username: Jordan_Howell
cooked: <p>I have a csv that contains a column of image file names, target labels and location of each file.  Just leaving tensorflow-keras side, I am admittedly a newbie to pytorch.  How do I code a dataloader to read the csv, and pull the images, randomly split off a test set and finally having a train and test set to pull in batches?</p>
<p>CSV columns are as such:<br>
location: directory of where the image sits (includes image file name)<br>
file name: image file name<br>
target: target variable consisting of 1 or 0</p>

username: Jordan_Howell
cooked: <p>I tried the folloiwng based off of <a href="https://jhui.github.io/2018/02/09/PyTorch-Data-loading-preprocess_torchvision/" rel="noopener nofollow ugc">https://jhui.github.io/2018/02/09/PyTorch-Data-loading-preprocess_torchvision/</a> but got an error. Code as below:</p>
<blockquote>
<p>class Roof_Dataset(Dataset):<br>
“”“Roof dataset.”“”</p>
<pre><code>def __init__(self, csv_file, root_dir, transform):
    """
    Args:
        csv_file (string): Path to the csv file with annotations.
        root_dir (string): Directory with all the images.
        transform (callable, optional): Optional transform to be applied
            on a sample.
    """
    
    self.roofs_frame = pd.read_csv(csv_file)
    self.root_dir = root_dir
    self.transform = transform

def __len__(self):
    return len(self.landmarks_frame)

def __getitem__(self, idx):
    img_name = Path(self.root_dir,
                            self.roof_frame.iloc[idx, 1])
    image = io.imread(img_name)
    landmarks = self.landmarks_frame.iloc[idx, 5]
    sample = {'image': image, 'landmarks': landmarks}

    if self.transform:
        sample = self.transform(sample)

    return sample
</code></pre>
</blockquote>
<p>I get  the following;</p>
<blockquote>
<hr>
<p>NameError                                 Traceback (most recent call last)<br>
 in <br>
----&gt; 1 class Roof_Dataset(Dataset):<br>
2     “”“Roof dataset.”“”<br>
3<br>
4     def <strong>init</strong>(self, csv_file, root_dir, transform):<br>
5         “”"</p>
<p>NameError: name ‘Dataset’ is not defined</p>
</blockquote>
<p>I’m also not sure, how I would have the data loader randomly cut off 20% for a test set. Am I headed in the wrong direction?</p>

username: JamesTrick
cooked: <p>Hi! Have you imported Dataset? You can do this by adding: <code>from torch.utils.data import Dataset</code>.</p>
<p>In terms of splitting off a validation set - you’ll need to do this outside the dataset. It’s probably easiest to use sklearns train_test_split. For example:</p>
<pre><code class="lang-auto">from sklearn.model_selection import train_test_split

train, val = train_test_split("full.csv", test_size=0.2)
train.to_csv("train.csv"), val.to_csv("val.csv")

train_dataset = Roof_dataset(csv_file="train.csv") . # Add any other params such as transforms here
val_dataset = Roof_dataset(csv_file="val.csv") # Again add any other params
</code></pre>

username: avinash08
cooked: <p>I am also facing same kind of difficulty. my csv file is different from images csv as my csv contains numerical data which will be feed to RNN model but i can’t find any documenatation to do that .</p>
<p>so as newbie I ask for your help . Thanks in advance.</p>

username: Jordan_Howell
cooked: <p>Hello <a class="mention" href="/u/avinash08">@avinash08</a>.  I was out of town.  When I get back to my desk on Monday, I’ll reply with what worked.</p>

username: Jordan_Howell
cooked: <p>I ended up combining into one csv with the location of my images as a column in my tabular data.  I have multiple images per record which means I take the mean prediction of one observation/mulitple images in the end. below is the code that worked on to pull the data.</p>
<p>class image_Dataset(Dataset):<br>
‘’’<br>
image class data set</p>
<pre><code>'''
def __init__(self, data, transform = None):
    '''
    Args:
    ------------------------------------------------------------
        data = dataframe
        image = column in dataframe with absolute path to the image
        label = column in dataframe that is the target classification variable
        numerical_columns =  numerical columns from data
        categorical_columns = categorical columns from data
        policy = ID variable
        
    '''
    self.image_frame = data
    self.transform = transform
    
def __len__(self):
    return len(self.image_frame)

def __getitem__(self, idx):
    if torch.is_tensor(idx):
        idx = idx.tolist()
     
    label = self.image_frame.loc[idx, 'target']
    pic = Path(self.image_frame.loc[idx,'location'])
    img = Image.open(pic)
    policy = self.image_frame.loc[idx, 'policy']
    #sample = {'image': img, 'policy': policy, 'label':label}
    numerical_data = self.image_frame.loc[idx, numerical_columns]
    
    numerical_data = torch.tensor(numerical_data, dtype = torch.float)

    if self.transform:
        image = self.transform(img)
        
    for category in categorical_columns:
        self.image_frame[category] = self.image_frame[category].astype('category')
        
        self.image_frame[category] = self.image_frame[category].astype('category').cat.codes.values
    
        
    categorical_data = self.image_frame.loc[idx, categorical_columns]
    categorical_data = torch.tensor(categorical_data, dtype = torch.int64)
        
    return image, label, policy, categorical_data , numerical_data</code></pre>

username: avinash08
cooked: <p>Thanx a lot . I was stuck in like for days just loading csv to model.</p>

username: Arnaud_Mal
cooked: <p>Hello,</p>
<p>I have a similar problem (<a href="https://discuss.pytorch.org/t/valueerror-expected-input-batch-size-4-to-match-target-batch-size-16/77516">link</a>) and I created a discussion for it.</p>
<p>I was able to create a CustomDataset that return an image and a label (both tensor). Then I pass them to the Dataloader, but then, we I get the Image and Target from the Dataloader in the BackPropagation, the size is not right.</p>
<p>The CustomDataset is:</p>
<pre><code class="lang-auto">class CustomDataset(Dataset):
    def __init__(self, csv_file, id_col, target_col, root_dir, sufix=None, transform=None):
        """
        Args:
            csv_file   (string):             Path to the csv file with annotations.
            root_dir   (string):             Directory with all the images.
            id_col     (string):             csv id column name.
            target_col (string):             csv target column name.
            sufix      (string, optional):   Optional sufix for samples.
            transform  (callable, optional): Optional transform to be applied on a sample.
        """
        self.data      = pd.read_csv(csv_file)
        self.id        = id_col
        self.target    = target_col
        self.root      = root_dir
        self.sufix     = sufix
        self.transform = transform

    def __len__(self):
        return len(self.data)

    def __getitem__(self, idx):
        # get the image name at the different idx
        img_name = self.data.loc[idx, self.id]
        
        # if there is not sufic, nothing happened. in this case sufix is '.jpg'
        if self.sufix is not None:
            img_name = img_name + self.sufix
        
        # it opens the image of the img_name at the specific idx
        image = Image.open(os.path.join(self.root, img_name))
        
        # if there is not transform nothing happens, here we defined below two transforms for train and for test
        if self.transform is not None:
            image = self.transform(image)
        
        # define the label based on the idx
        #label = self.data.loc[idx, self.target].values
        #label = torch.from_numpy(label.astype(np.int8))
        #label = label.squeeze(-1)
        
        #Test second option
        
        label_test = self.data.iloc[idx, 1:5].values.astype('float32')
        
        return image, label_test
</code></pre>
<p>and the data_transforms and params are as below</p>
<pre><code class="lang-auto">data_transforms = {
    'train': transforms.Compose([
        transforms.Resize(256),
        transforms.CenterCrop(224),
        transforms.RandomHorizontalFlip(),
        transforms.ToTensor(),
    ]),
    'test': transforms.Compose([
        transforms.Resize(256),
        transforms.CenterCrop(224),
        transforms.ToTensor(),
    ])
}

params = {
    'id_col':     'image_id',  
    'target_col': ['healthy', 'multiple_diseases', 'rust', 'scab'],
    'sufix':      '.jpg',
    'transform':  data_transforms['train']
}

train_dataset = CustomDataset(csv_file=data_dir+'train.csv', root_dir=data_dir+'images', **params)
train_loader = DataLoader(train_dataset, batch_size=4, shuffle=True)
</code></pre>
<p>As you see, the target is four categories where the image is identified. Like this</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/1/6/16cef08bb7677ae5c22be2409ee0521e249aec6e.jpeg" alt="pdFrame" data-base62-sha1="3fLRY4wJGJ4yOapmVr4BUfNCV4i" width="371" height="174"></p>
<p>My issue is when I do the backpropagation with the dataloader, I have the wrong target size ([16]) instead of ([4]).</p>
<p>The training is like this</p>
<pre><code class="lang-auto">
def train2(n_epochs, loaders, model, optimizer, criterion):
    """returns trained model"""
    # initialize tracker for minimum validation loss
    valid_loss_min = np.Inf 
    
    for epoch in range(1, n_epochs+1):
        # initialize variables to monitor training and validation loss
        train_loss = 0.0
        valid_loss = 0.0
        
        ###################
        # train the model #
        ###################
        model.train()
        for idx, (data, target) in enumerate(loaders):

            ## find the loss and update the model parameters accordingly
            ## record the average training loss, using something like
            ## train_loss = train_loss + ((1 / (batch_idx + 1)) * (loss.data - train_loss))
            optimizer.zero_grad()
            # forward pass: compute predicted outputs by passing inputs to the model
            output = model(data)
            print(idx)
            target.view(-1)
            print(target.shape)
            target = target.long()
            loss = criterion(output, target)
            # backward pass: compute gradient of the loss with respect to model parameters
            loss.backward()
            # perform a single optimization step (parameter update)
            optimizer.step()
            #update training loss
            train_loss += loss.item()*data.size(0)
            
        # calculate average losses
        train_loss = train_loss/len(loaders.sampler)
        # print training/validation statistics 
        print('Epoch: {} \tTraining Loss: {:.6f} \tValidation Loss: {:.6f}'.format(
            epoch, 
            train_loss,
            ))
            
    # return trained model
    return model
</code></pre>
<p>This is the link to the <a href="https://github.com/ArnaudMallet/Plant_Patho.git" rel="noopener nofollow ugc">github</a> so I can track my progress so you can have the full picture.</p>
<p>Any ideas, suggestions?</p>

username: ptrblck
cooked: <p>Which batch size are you using for the <code>DataLoader</code>s and could you please print the shape of <code>target</code> before the <code>view(-1)</code> operation?</p>

username: Arnaud_Mal
cooked: <p>The batchsize is 4:</p>
<p><code>train_loader = DataLoader(train_dataset, batch_size=4, shuffle=True)</code></p>
<p>The shape of the Target is <code>torch.Size([4, 4])</code></p>
<p>Originally I wanted to use something like:</p>
<pre><code class="lang-auto">abel = pd.read_csv(csv_file).loc[idx, ['healthy', 'multiple_diseases', 'rust', 'scab']].values
label = torch.from_numpy(label.astype(np.int8))
</code></pre>
<p>But this doesn’t work neither</p>
<p>Eventually, I found another way to do it (but I am not sure whether it is right, as I am still not able to have decent predictions on the test data), in the CustomDataset I replace the label related operation by:</p>
<pre><code class="lang-auto">label = self.data.iloc[idx, 1:5].values.astype('int64')
label = np.argwhere(label ==1)
label = label.item(0)
label = torch.tensor(label)
</code></pre>
<p>The label shape is 'torch.Size([ ]), which gives me a tensor of the shape torch.Size([4]) from the Dataloader. This way the training works with a simple:</p>
<pre><code class="lang-auto">for idx, (data, target) in enumerate(loaders):

            ## find the loss and update the model parameters accordingly
            ## record the average training loss, using something like
            ## train_loss = train_loss + ((1 / (batch_idx + 1)) * (loss.data - train_loss))
            optimizer.zero_grad()
            output = model(data)
            loss = criterion(output, target)
</code></pre>
<p>but I am not sure it is the proper way to do it, and I would like to be sure to understand the reason why the previous solution didn’t work.</p>
<p>Thanks</p>

username: ptrblck
cooked: <p>Thanks for the update!<br>
The previous solution wasn’t working, since your target was one-hot encoded (<code>[ 4, 4]</code>) and you’ve flattened it to a tensor of <code>[16]</code>, which created the shape mismatch.<br>
<code>nn.CrossEntropyLoss</code> expects the target to contain the class indices, which can be created via <code>torch.argmax(one_hot_target, dim=1)</code>. Your approach with numpy also seems to be correct.</p>

username: saba
cooked: <p>Hi Ptrblck,</p>
<p>What is the best way to save the results (numbers , vectors, matrix) with the same type ?and reload them again ?</p>

username: ptrblck
cooked: <p>Assuming this data is stored as PyTorch tensors, you can use <code>torch.save</code> and <code>torch.load</code>.<br>
If that’s not the case, you could either use Python directly to store these values or e.g. numpy.</p>

username: saba
cooked: <p>Hi Ptrblck,</p>
<p>I am loading the training data saved with .csv. It is not the first time that I use torch.load (…csv). on the GPU. However, this time it gave me this error  :zip archive (did you mean to use torch.jit.load()?)<br>
.</p>
<p>On another code the same command is working without any error. but on other it gave me this error and when I use the recommended load command it gave me another strange error.</p>
<p>I wondering if you know what is teh best way of loading for this problem?</p>
<p>Many thanks</p>

username: ptrblck
cooked: <p>I guess the error might be raised if you are mixing different PyTorch versions or <code>torch.save</code> with <code>torch.jit.save</code>. Could this be the case?</p>

username: saba
cooked: <p>i don’t think so. because the other code is running without error.</p>
<p>can I save my tensor data by using the following command use no (.csv) at the end?</p>
<pre><code class="lang-auto">Path=root_diR1+'/'+'Itr='+str(ii)+'Gen='+str(GEN)+'Num='+str(Num)+'Coef='+str(CoeBL)

torch.save(TrainPatchGAN,Path,_use_new_zipfile_serialization=False))</code></pre>

username: ptrblck
cooked: <p>Yes, you can store tensors using this command. If you are struggling with the csv file, feel free to post an executable code snippet to reproduce this issue.</p>