Create DataLoader from list of NumPy arrays

lorenzo_fabbri: <p>I’m trying to build a simple CNN where the input is a list of NumPy arrays and the target is a list of real numbers (regression problem).</p>
<p>I’m stuck when I try to create the DataLoader.</p>
<p>Suppose <code>Xp_train</code> and <code>yp_train</code> are two Python lists that contain NumPy arrays. Currently I’m using the following code:</p>
<pre><code class="lang-auto">tensor_Xp_train = torch.stack([torch.Tensor(el) for el in Xp_train])
tensor_yp_train = torch.stack([torch.Tensor(el) for el in yp_train])
dataset_p_train = TensorDataset(tensor_Xp_train, tensor_yp_train)
loader_p_train = DataLoader(dataset_p_train)
</code></pre>
<p>The problem is that it’s not working since <code>Xp_train</code> is a list of 149798 arrays.</p>
<p>Is there a way to create a DataLoader without loading everything into memory?</p>

ptrblck: <p>You could create a <code>Dataset</code> and load the data lazily.<br>
However, if you have already loaded the numpy arrays, they should apparently fit into your RAM.<br>
Try to use <code>torch.from_numpy</code> to reuse the underlying memory and to avoid a copy.</p>

ayrts: <p>Sorry for the necrobump, but I am also having a similar issue. I have a huge list of numpy arrays (&gt;100, 000), and I am trying to create a custom <code>Dataset</code> that imitates <code>ImageFolder</code> with two classes (fake and real).</p>
<p>However, I still run out of memory even before training starts. How can I load the data lazily? Below is my code that implements my custom dataset. I call it with <code>CustomDataset(args.data_root)</code>, where <code>args.data_root</code> is the root directory of my dataset, then I pass it to the vanilla <code>DataLoader</code>. Any help is appreciated. Thanks!</p>
<pre><code class="lang-auto">class CustomDataset(Dataset):
    def __init__(self, data_root, transform=None):
        fake = join(data_root, 'fake')
        real = join(data_root, 'real')
        data=[np.load(join(fake, array)) for array in os.listdir(fake)]
        data.extend([np.load(join(real, array)) for array in os.listdir(real)])

        target=[np.zeros(1, dtype=np.long) for i in range(len([array for array in os.listdir(fake) if os.path.isfile(array)]))]
        target.extend([np.ones(1, dtype=np.long) for i in range(len([array for array in os.listdir(real) if os.path.isfile(array)]))])
        target=[target[i][0] for i in range(len(target))]
        self.data = torch.from_numpy(data)
        self.target = torch.from_numpy(target)
        self.transform = transform
</code></pre>

ptrblck: <p>To load data lazily, you would have to move the actual loading into <code>Dataset.__getitem__</code>, while you are apparently preloading the complete dataset in the <code>__init__</code> method:</p>
<pre><code class="lang-python">data=[np.load(join(fake, array)) for array in os.listdir(fake)]
data.extend([np.load(join(real, array)) for array in os.listdir(real)])
</code></pre>
<p>While this is easy to do, if you are dealing with samples stored in unique files (e.g. images), you might need to implement more logic, e.g. if each numpy array contains multiple samples.</p>
<p>I.e. you could load each numpy array and return it completely. This approach would basically multiply your <code>batch_size</code> (passed to the <code>DataLoader</code>) with the number of samples per loaded array.<br>
Also, the <code>shuffle</code> option would only shuffle the numpy files, not the samples directly.</p>

ayrts: <p>Thanks for the reply Patrick. Each numpy array only contains one sample. It turns out <code>DatasetFolder</code> is exactly what I need because I basically needed a generic implementation of <code>ImageFolder</code>. I implemented it like this:</p>
<pre><code class="lang-auto">def npy_loader(path):
                sample = torch.from_numpy(np.load(path))
                return sample
train_set = DatasetFolderWithPaths(root=args.data_root, loader=npy_loader, extensions='.npy', transform= transform)
</code></pre>

jdhmr: <p>Hi Patrick, sorry for the revival of an old post.</p>
<p>I am using a huge dataset (8 GB) of numpy arrays.</p>
<p>I have tried an <code>lmdb</code> dataset approach, which involved conversion of the arrays into <code>.jpg</code> formats, I haven’t been able to reproduce my previous results on arrays.</p>
<p>The solutions here are ideal, but would you say it is a bad idea to load <code>.npy</code> arrays every minibatch? It seems people don’t tend to do this!</p>

ptrblck: <p>If I understand your use case correctly, you are loading a <code>.npy</code> file for each sample? If so, I don’t think it’s a bad idea or do you have any specific concerns?<br>
I think the usual approach of loading image types (e.g. JPEG) has given you the advantage to apply image transformations on the samples directly, but since <code>torchvision.transforms</code> now also supports transformations on tensors you could also load the arrays/tensors directly.</p>