Input numpy ndarray instead of images in a CNN

carioka_88: <p>Hello,</p>
<p>I am kind of new with Pytorch.</p>
<p>I would like to run my CNN with some ordered datasets that I have.<br>
I have n-dimensional arrays, and I would like to pass them like the input dataset.<br>
Is there any way to pass it with torch.utils.data.DataLoader?<br>
Or how can I transform the n-dimensional array into a DataLoader object?</p>
<p>For example, right now I have something like this for images:</p>
<pre><code class="lang-auto">image_datasets = {x: datasets.ImageFolder(os.path.join(data_dir, x),
                                          data_transforms[x])
                  for x in ['train', 'val']}
dataloaders = {x: torch.utils.data.DataLoader(image_datasets[x], batch_size=4,
                                             shuffle=True, num_workers=4)
</code></pre>
<p>But what I have is something like:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; x
array([[[ 0,  1,  2],
        [ 3,  4,  5],
        [ 6,  7,  8]],
       [[ 9, 10, 11],
        [12, 13, 14],
        [15, 16, 17]],
       [[18, 19, 20],
        [21, 22, 23],
        [24, 25, 26]],
        .......
        ])
</code></pre>
<p>At the end I would like to treat this n-dimensional array, like the pixel values of one image.<br>
Is there a way to pass this values to the CNN like the dataloaders?</p>

ptrblck: <p>You could create a <code>Dataset</code>, and load and transform your arrays there.<br>
Here is a small example:</p>
<pre><code class="lang-auto">import torch
from torch.utils.data import Dataset, DataLoader

import numpy as np


class MyDataset(Dataset):
    def __init__(self, data, target, transform=None):
        self.data = torch.from_numpy(data).float()
        self.target = torch.from_numpy(target).long()
        self.transform = transform
        
    def __getitem__(self, index):
        x = self.data[index]
        y = self.target[index]
        
        if self.transform:
            x = self.transform(x)
        
        return x, y
    
    def __len__(self):
        return len(self.data)


numpy_data = np.random.randn(100, 3, 24, 24)
numpy_target = np.random.randint(0, 5, size=(100))

dataset = MyDataset(numpy_data, numpy_target)
loader = DataLoader(
    dataset,
    batch_size=10,
    shuffle=True,
    num_workers=2,
    pin_memory=torch.cuda.is_available()
)

for batch_idx, (data, target) in enumerate(loader):
    print('Batch idx {}, data shape {}, target shape {}'.format(
        batch_idx, data.shape, target.shape))
</code></pre>

jain26iitd: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Thanks for the code.<br>
I used the code as below to create a custom dataloader.</p>
<pre><code class="lang-auto"># Creating custom Dataset classes

class MyDataset(Dataset):
    def __init__(self, data, target, transform=None):
        self.data = torch.from_numpy(data).float()
        self.target = torch.from_numpy(target).long()
        self.transform = transform
        
    def __getitem__(self, index):
        x = self.data[index]
        y = self.target[index]
        
        if self.transform:
            x = self.transform(x)
            
        return x, y
    
    def __len__(self):
        return len(self.data)
</code></pre>
<pre><code class="lang-auto">numpy_data = np.random.randn(100,3,224,224) # 10 samples, image size = 224 x 224 x 3
numpy_target = np.random.randint(0,5,size=(100))

dataset = MyDataset(numpy_data, numpy_target)
loader = DataLoader(dataset, batch_size=1, shuffle=True, num_workers=2, pin_memory=False)  # Running on CPU
</code></pre>
<p>Till this part everything is fine.<br>
The below mentioned part of the code is taking too much time to run.<br>
Any help on this??</p>
<pre><code class="lang-auto">for batch_idx, (data, target) in enumerate(loader):
    print('Batch idx {}, data shape {}, target shape {}'.format(batch_idx, data.shape, target.shape))
</code></pre>

ptrblck: <p>How long does your code take?<br>
It’s running on my machine basically immediatelly.</p>

jain26iitd: <p><a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p>waited for around 20 minutes with no luck!</p>

ptrblck: <p>Try to set <code>num_workers=0</code> and run it again.<br>
If that’s working, could you post your PyTorch version, OS, and how you’ve installed PyTorch?</p>

jain26iitd: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a></p>
<p><strong>It worked!</strong></p>
<p>OS : Windows 10 (64 bit)<br>
PyTorch Version : 0.4.1<br>
Python : 3.6.6</p>
<p>PyTorch installed using</p>
<pre><code class="lang-auto">pip3 install http://download.pytorch.org/whl/cpu/torch-0.4.1-cp36-cp36m-win_amd64.whl
pip3 install torchvision
</code></pre>

ptrblck: <p>Thanks for the info.<br>
Since you are using Windows, could you add the if-clause protection as described <a href="https://pytorch.org/docs/stable/notes/windows.html#usage-multiprocessing">here</a>?</p>
<p>Let me know, if this approach works with multiple workers.</p>

NIRVANALAN: <p>Hey. I ran into the same problem, loading attribute labels from large(1 GB+) numpy array saved as .npy on the server. It seems that I cannot load them using multiple workers(num_worker&gt;0 does not work and the code just hang there, waiting).When I set num_worker=0,  each 100 iteration costs 100 seconds on loading&amp; pre processing data and 50 seconds for training(forward backward step), which is really slow compared to not loading those numpy labels. I got stuck on this problem and eagerly looking for solutions. Would you give me some advice?</p>
<p>Sincerely.</p>

ptrblck: <p>Are you loading this large file once in your <code>__init__</code>?<br>
If so, <code>num_workers=0</code> should pre-load the data once and just slice it in <code>__getitem__</code>.<br>
Using multiple workers might not be the best idea, as each worker will load the whole dataset, while the <code>__getitem__</code> should be quite fast compared to the <code>__init__</code>.</p>
<p>Could you post the <code>Dataset</code> implementation?<br>
Maybe using <a href="https://github.com/ptrblck/pytorch_misc/blob/master/shared_array.py" rel="nofollow noopener">shared arrays</a> might work better?</p>

NIRVANALAN: <p>Thanks for replying so quickly <img src="https://discuss.pytorch.org/images/emoji/apple/grin.png?v=9" title=":grin:" class="emoji" alt=":grin:"></p>
<p>My implementation is as follows:</p>
<pre><code class="lang-auto">	def __init__(self, image_list_path, height, width, attribute_list_path=None, use_attribute=False, root=None,
	             transform=None):
		self.height = height
		self.width = width
		self.root = root
		self.transform = transform
		self.image_list = []
		self.attribute_list_path = attribute_list_path
		self.num_classes = 0
		with open(image_list_path, 'r') as f:
			for line in f:
				x, y = line.split()
				self.image_list.append((x, int(y)))
				self.num_classes = max(self.num_classes, int(y) + 1)
		self.attribute_list = list(np.load(attribute_list_path))  # attribute numpy array
	
	def __len__(self):
		return len(self.image_list)
	
	def __getitem__(self, index):
		filename = self.image_list[index][0]
		if self.root is not None:
			filename = osp.join(self.root, self.image_list[index][0])
		if not osp.exists(filename):
			raise IOError('File is not exist: ', filename)
		
		try:
			img = pil_loader(filename)
		except:
			raise IOError('File error: ', filename)
		img = cv2.resize(img, (self.width, self.height))
		img = torchvision.transforms.ToPILImage()(img)
		
		if self.transform is not None:
			img = self.transform(img)


		self.attribute_list[index] = dict(sorted(self.attribute_list[index].items(), key=lambda x:x[0]))
		value_list = list(self.attribute_list[index].values())  # does not support dict_values
		attr_labels = [list(v.values()) for v in value_list]

		return img, attr_labels, self.image_list[index][1]  # image label ID
	
	# else:
	# 	return img, self.image_list[index][1]  # image &amp; ID
	
	def get_num_classes(self):
		return self.num_classes
</code></pre>

NIRVANALAN: <p>In general, I want to load attribute label from a large numpy file. This file is loaded in <strong>init</strong> and the labels are returned in get_item(). I wonder if multiple workers are supported under this situation…</p>

ptrblck: <p>The code for loading the image paths looks alright, although you could also pre-create the lists and just pass it to your <code>Dataset</code> instead of re-creating it in the <code>__init__</code>.<br>
The same applies for <code>attribute_list_path</code>.<br>
Note that the <code>Dataset</code> will be re-created if you are using multiple workers for each epoch, so that each worker will reload the large numpy array.<br>
Just preload it outside of your <code>Dataset</code> and pass it as an argument to it.</p>

NIRVANALAN: <p>Thanks ! That is really a helpful advice and I will try it soon. By the way, do you mean the reason that my code just stuck is because of loading too much data into the memory? Will multi workers help if I preload the numpy array and just pass an argument to my Dataset, will it cause multi-processing resource competition problem if several workers want to read the same data(as reference) ?</p>
<p>Your reply is really helpful and thanks again here <img src="https://discuss.pytorch.org/images/emoji/apple/laughing.png?v=9" title=":laughing:" class="emoji" alt=":laughing:"></p>

ptrblck: <p>I guess your code might just hang, if the loading takes so long or if you run out of memory.<br>
However, I think I might be mistaken and also the pre-loaded dataset might be copied using multiple workers.<br>
The shared array approach might speed thing up.</p>

NIRVANALAN: <p>Thanks. I will try pre-loading method and shared array approach soon, and let you know which approach works better</p>
<p>Thanks again for your ardent and helpful reply.<img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=9" title=":smile:" class="emoji" alt=":smile:"></p>

Rui_Zhang: <p>Hi,</p>
<p>As I linked in <a href="https://discuss.pytorch.org/t/solved-make-sure-that-pytorch-using-gpu-to-compute/4870/22">another ticket</a>, I found that this implementation is lack of vectorisation. When one retrieves data in <code>loader</code>, <code>MyDataset.__getitem__</code> will be called millions of times. This becomes a bottleneck of my training on GPU. In Keras, we know that larger batch_size will reduce the training time; however here, batch_size will have small effect on the training time due to the loop over the training points. Is there any suggestion to avoid this?</p>

ptrblck: <p>Each worker in your <code>DataLoader</code> will create the next batch in the background by calling <code>__getitem__</code> to load the corresponding sample.<br>
I’m not sure, if there is a library to load e.g. images in a batch way.<br>
As explained in the other topic: using multiple workers might speed your data loading up, if your hard disk is sufficiently fast.<br>
Have a look at <a href="https://discuss.pytorch.org/t/how-to-prefetch-data-when-processing-with-gpu/548/19">this post</a> for more information.</p>

Caroline_Barcelos: <p>Hi,</p>
<p>I’m using the custom dataset class proposed but I ran into the problems when applying transform to the converted numpy array.<br>
The transform I’m trying to apply is:</p>
<p>transforms.Compose([<br>
transforms.ToPILImage(),<br>
transforms.Resize((224, 224)),<br>
transforms.ToTensor(),<br>
transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])<br>
])</p>
<p>the error I get is “RuntimeError: output with shape [1, 224, 224] doesn’t match the broadcast shape [3, 224, 224]”<br>
Would you give me some advice of how my transform should be in order to apply it to the converted numpy array?</p>

axki: <p>Hi!</p>
<p>When comparing the transform you defined and the error you got, it seems like your input is only one layer while your transform expects three layers (looking at the normalization-part):</p>
<aside class="quote no-group" data-username="Caroline_Barcelos" data-post="19" data-topic="18797">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/caroline_barcelos/48/23435_2.png" class="avatar"> Caroline_Barcelos:</div>
<blockquote>
<p>transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])</p>
</blockquote>
</aside>
<p>You should change your normalization so that it only has one value for the mean and one for standard deviation. Then the transform should be valid and you can just apply it to your converted numpy array <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>