Loading .npy files using torchvision

jingjing: <p>Dear all,</p>
<p>I am trying to train my own Resnet model using .npy format files.</p>
<p>I am wondering that are there any functions like  torchvision.datasets.ImageFolder that can load .npy files in a folder and label these numpy array with their folder name?</p>

albanD: <p>Hi,</p>
<p>No such dataset exist at the moment.<br>
I guess you can load the files, convert these to torch tensor and create a <a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.TensorDataset">TensorDataset</a> from them.</p>

ptrblck: <p>Alternatively to <a class="mention" href="/u/alband">@albanD</a>’s solution, you could also use <a href="https://pytorch.org/docs/stable/torchvision/datasets.html#datasetfolder">DatasetFolder</a>, which basically is the underlying class of <code>ImageFolder</code>.<br>
Using this class you can provide your own files extensions and <code>loader</code> to load the samples.</p>
<pre><code class="lang-python">def npy_loader(path):
    sample = torch.from_numpy(np.load(path))
    return sample
    

dataset = datasets.DatasetFolder(
    root='PATH',
    loader=npy_loader,
    extensions=['.npy']
)
</code></pre>
<p>If you want to use transformations, you would need to convert the sample tensors to <code>PIL.Images</code> in your loader.</p>

jingjing: <p>Thank you ptrblck! That did work!</p>

Tanya_Boone: <p>Hello ptrblck…I am facing the same problem but I am getting some errors…</p>
<p>RuntimeError: Found 0 files in subfolders of: cifar-100//cifar-100-python//pylearn2_gcn_whitened//train//<br>
Supported extensions are: .npy</p>
<p>so, if I change the code:</p>
<pre><code class="lang-auto">path='cifar-100/cifar-100-python/pylearn2_gcn_whitened/train/train.npy'
sample_train = torch.from_numpy(np.load(path))

    
print(len(sample_train)) #50000

path='cifar-100/cifar-100-python/pylearn2_gcn_whitened/test/test.npy'
sample_test = torch.from_numpy(np.load(path))

    
print(len(sample_test)) #10000

</code></pre>
<p>I can get them as tensors on each variables, but I dont know how to transform them to type dataset.</p>
<p>note: this datasets were transformed from the original cifar100 but using the library pylearn2</p>

ptrblck: <p>The use case in the original question was that instead of images in subfolders, <code>.npy</code> files were located in these subfolders.<br>
Since <code>ImageFolder</code> only looks for image files, I suggested to use <code>DatasetFolder</code> with a custom loader.</p>
<p>However, your use case is different, since you have a single <code>.npy</code> file with apparently all images stored inside of it.</p>
<p>In that case, you should stick to <a class="mention" href="/u/alband">@albanD</a>’s solution and use <code>TensorDataset(torch.from_numpy(np.load(path)))</code>.</p>

Tanya_Boone: <p>Thanks ptrblck</p>
<p>I solved that with saving them in a pt file and then creating a Custom Dataset</p>
<pre><code class="lang-auto">__author__ = 'mangate'

from six.moves import cPickle as pickle
import numpy as np
import os
import fnmatch
import sys
#import matplotlib.pyplot as plt
from pylearn2.datasets.cifar100 import CIFAR100
import torch
"""
This file opens the CIFAR100 data after whitening and ZCA made by 'process_cifar_100_data' script
which uses pylearn2 library
This file also re-arragne the data so it can enter a nueral net properly
"""

image_size = 32
num_channels = 3
num_classes = 100
pixel_depth = 255.0

def unpickle(file):
    #import pickle
    #import cPickle
    import _pickle as cPickle
    fo = open(file, 'rb')
    dict = cPickle.load(fo)
    fo.close()
    return dict

def rearrange_data(data,labels):
    #data = np.cast['float32'](data)
    #data = data / 255.
    #data = data - data.mean(axis=0)
    images = np.ndarray(shape=(len(labels),image_size,image_size,num_channels), dtype=np.float32)
    labels_out = np.zeros(shape=(len(labels),num_classes),dtype=np.float32)
    max = len(labels)
    for i in range(max):
        images[i] = np.reshape(data[i],(3,32,32)).transpose(1,2,0)
        #plt.imshow(images[i])
        #plt.show()
        labels_out[i][labels[i]]=1.0
    return images,labels_out

def process_data(file_name):
    data = unpickle(file_name)
    # images = data['data']
    # labels = data['fine_labels']
    images = np.asarray(data.X)
    labels = np.asarray(data.y)
    images,labels = rearrange_data(images,labels)
    return images,labels

def get_data():
    ROOT_FOLDER ="D:\\Neural_Nets\\ELU_NETWORK\\cifar100\\cifar-100-python\\pylearn2_gcn_whitened\\"
    train_images, train_labels = process_data(ROOT_FOLDER+'train.pkl')
    test_images,test_labels = process_data(ROOT_FOLDER+'test.pkl')
    # ROOT_FOLDER = '/cs/img/mangate/thesis/Cifar-100/cifar-100-python/'
    #
    # train_images, train_labels = process_data(ROOT_FOLDER+'train')
    # test_images,test_labels = process_data(ROOT_FOLDER+'test')
    print ('Train Date shape is',train_images.shape, 'and labels is',train_labels.shape)
    print ('Test Date shape is',test_images.shape, 'and labels is',test_labels.shape)
    torch.save(train_images,'train_images.pt')
    torch.save(train_labels,'train_labels.pt')
    torch.save(test_images,'test_images.pt')
    torch.save(test_labels,'test_labels.pt')
    return  train_images, train_labels, test_images,test_labels

get_data()
</code></pre>

Muhammad_Saqlain: <p>Thank you <a class="mention" href="/u/ptrblck">@ptrblck</a> for providing a solution to this problem.  I was facing the same problem, luckily I found this post. But when I run this code I face another Error. Can you please help me to solve it.</p>
<p>TypeError: endswith first arg must be str or a tuple of str, not list</p>

ptrblck: <p>Based on the error message it seems you are passing a <code>list</code> to <code>endswith</code> as the first argument, which is not supported, so you would need to pass either a <code>str</code> or a <code>tuple</code> of <code>str</code>.</p>
<pre><code class="lang-python">text = 'lala'
text.endswith('a') # works
text.endswith(('a', 'la')) # works
text.endswith(['a', 'la'])
&gt; TypeError: endswith first arg must be str or a tuple of str, not list
</code></pre>

Muhammad_Saqlain: <p>Thank you <a class="mention" href="/u/ptrblck">@ptrblck</a> for your quick reply, but I am still confused with the same Error. Below is the pic of my code. Training folder contains three subfolders with labels name and each subfolder contains various .npy files. Where do I should change my code or update any library or folder? How can I load this data?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/e/bea62dd6a03dfeabe04466080d8bbc2c68f9b646.png" data-download-href="https://discuss.pytorch.org/uploads/default/bea62dd6a03dfeabe04466080d8bbc2c68f9b646" title="Screenshot from 2021-07-27 19-25-05"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/e/bea62dd6a03dfeabe04466080d8bbc2c68f9b646_2_690x399.png" alt="Screenshot from 2021-07-27 19-25-05" data-base62-sha1="rcyKTuqtwEiyS3KpunAakUzQM86" width="690" height="399" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/e/bea62dd6a03dfeabe04466080d8bbc2c68f9b646_2_690x399.png, https://discuss.pytorch.org/uploads/default/optimized/3X/b/e/bea62dd6a03dfeabe04466080d8bbc2c68f9b646_2_1035x598.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/b/e/bea62dd6a03dfeabe04466080d8bbc2c68f9b646_2_1380x798.png 2x" data-dominant-color="282829"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2021-07-27 19-25-05</span><span class="informations">1454×841 81.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>Use <code>extensions='.npy'</code> or <code>extensions=('.npy')</code>, as the error is raised by <code>DatasetFolder</code>.</p>

Muhammad_Saqlain: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a>. Yes, now it works.</p>

devsgnr: <p>Hey <a class="mention" href="/u/ptrblck">@ptrblck</a> - I have a <code>FileNotFoundError</code> error, what is it that I am missing?</p>
<hr>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/4/945bb10e4ad974e9ffd5048eed92c0d7d3498e5e.png" data-download-href="https://discuss.pytorch.org/uploads/default/945bb10e4ad974e9ffd5048eed92c0d7d3498e5e" title="Screenshot 2023-02-06 at 00.30.30"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/4/945bb10e4ad974e9ffd5048eed92c0d7d3498e5e_2_690x310.png" alt="Screenshot 2023-02-06 at 00.30.30" data-base62-sha1="lar74jun2W1Rt8lLKG3IxOd3AS2" width="690" height="310" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/4/945bb10e4ad974e9ffd5048eed92c0d7d3498e5e_2_690x310.png, https://discuss.pytorch.org/uploads/default/optimized/3X/9/4/945bb10e4ad974e9ffd5048eed92c0d7d3498e5e_2_1035x465.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/9/4/945bb10e4ad974e9ffd5048eed92c0d7d3498e5e.png 2x" data-dominant-color="151822"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2023-02-06 at 00.30.30</span><span class="informations">1104×496 46.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p><code>root</code> should point to the root directory containing subfolders with the actual data for each corresponding class. <code>FileNoFoundError</code>s are raised if e.g. <code>np.load</code> cannot load the specified file.</p>

devsgnr: <p>Thank you very much for the nodge <a class="mention" href="/u/ptrblck">@ptrblck</a> - I checked the <a href="https://pytorch.org/vision/stable/generated/torchvision.datasets.DatasetFolder.html#torchvision.datasets.DatasetFolder.find_classes" rel="noopener nofollow ugc">documentation</a> and apparently, I have to use the naming convention of <code>class_x</code> and <code>class_y</code> for it to work. <img src="https://discuss.pytorch.org/images/emoji/apple/blush.png?v=12" title=":blush:" class="emoji" alt=":blush:" loading="lazy" width="20" height="20"></p>

DAdak: <p>No It did not work for me. I made following changes that worked:</p>
<p>class NPYDatasetFile(Dataset):<br>
“”“<span class="mention">@Deba</span>.”“”</p>
<pre><code>def __init__(self,  root_dir, transform=None):
    """
    Args:
        
        root_dir (string): Directory with all the images.
        transform (callable, optional): Optional transform to be applied
            on a sample.
    """
    self.csv_file="face_landmarks.csv" ###must have
    self.landmarks_frame = pd.read_csv(root_dir + self.csv_file)
    self.root_dir = root_dir
    self.transform = transform

def __len__(self):
    return len(self.landmarks_frame)

def __getitem__(self, idx):
    if torch.is_tensor(idx):
        idx = idx.tolist()

    img_name = os.path.join(self.root_dir,
                            self.landmarks_frame.iloc[idx, 0])
    image = np.load(img_name)
    sample=np.zeros((1,4*nside,3*nside)) ###for my data shape ##
    
    
    sample[0] = image#{'image': image}

    if self.transform:
        sample = self.transform(sample)

    return sample
</code></pre>
<p>transformed_dataset =  NPYDatasetFile(root_dir=path)<br>
dataloader = DataLoader(transformed_dataset, batch_size=4,<br>
shuffle=True, num_workers=4)</p>

ptrblck: <p>It’s unclear what exactly did not work for you using the <code>DatasetFolder</code> approach using a custom <code>loader</code> explained in my post, as you are not even using this class. Also your code is neither executable nor does it show any errors.</p>