How to load specific number of images from datasets.ImageFolder

username: lanka
cooked: <p>hi,<br>
i need limit number of load images to <code>all_deta = datasets.ImageFolder(path)</code><br>
example folder contains 1000 images(including all subfolders) i want ot load 800 among them only how can i do that , later on i want use<code> iter</code> and <code>next</code> as well.</p>

username: ptrblck
cooked: <p>You could wrap the <code>all_deta</code> object into <code>torch.utils.data.Subset</code> and pass the wanted 800 indices to it.</p>

username: lanka
cooked: <p>i did it , like<br>
<code>dataset_subset = torch.utils.data.Subset(all_data, np.random.choice(len(all_data), 800, replace=False))</code></p>
<pre><code class="lang-auto">train_data, test_data = random_split(dataset_subset, [train_data, test_data])
train_loader = DataLoader(train_data, batch_size=batch_size, shuffle=True)
</code></pre>
<p>problem is…its not iterable through</p>
<pre><code class="lang-auto">trainiter = iter(train_loader)
features, labels = next(trainiter)
</code></pre>
<p>error was<br>
<code>TypeError: default_collate: batch must contain tensors, numpy arrays, numbers, dicts or lists; found &lt;class 'PIL.Image.Image'&gt;</code> how can we modify iter and next according to imagefolder object</p>

username: ptrblck
cooked: <p>Your <code>ImageFolder</code> is missing the <code>transform</code> to return the supported classes and returns <code>PIL.Image</code>s instead. Add <code>transform=torchvision.transforms.ToTensor()</code> and it should most likely work.</p>

username: lanka
cooked: <p>i did that also,<br>
image_transforms = {</p>
<pre><code>'train':torchvision.transforms.Compose([
        torchvision.transforms.Resize(size=image_size),
        torchvision.transforms.RandomHorizontalFlip(),
        torchvision.transforms.RandomCrop(size=image_size),
        torchvision.transforms.ToTensor(),
        torchvision.transforms.Normalize(mean=(0.485, 0.456, 0.406),
                                         std=(0.229, 0.224, 0.225))
</code></pre>
<p>issue is same,</p>

username: ptrblck
cooked: <p>Could you post a minimal, executable code snippet to reproduce the issue, please?</p>

username: lanka
cooked: <p>import numpy as np<br>
import torchvision<br>
import torch<br>
from torch.utils.data import Dataset, DataLoader, random_split, sampler<br>
import torch.optim as optim<br>
import torch.nn as nn<br>
import torch.nn.functional as F<br>
from torchvision import models, datasets, models<br>
import random<br>
import os</p>
<p>batch_size = 16<br>
image_size = 299</p>
<p>image_transforms = {</p>
<pre><code>'train':torchvision.transforms.Compose([
        torchvision.transforms.Resize(size=image_size),
        torchvision.transforms.RandomHorizontalFlip(),
        torchvision.transforms.RandomCrop(size=image_size),
        torchvision.transforms.ToTensor(),
        torchvision.transforms.Normalize(mean=(0.485, 0.456, 0.406),
                                         std=(0.229, 0.224, 0.225))
]),
'val':torchvision.transforms.Compose([
        torchvision.transforms.Resize(size=image_size),
        torchvision.transforms.CenterCrop(size=image_size),
        torchvision.transforms.ToTensor(),
        torchvision.transforms.Normalize(mean=(0.485, 0.456, 0.406),
                                         std=(0.229, 0.224, 0.225))
    ])
</code></pre>
<p>}<br>
path=‘D:/imgs/’<br>
all_data = datasets.ImageFolder(root=path)</p>
<p>dataset_subset = torch.utils.data.Subset(all_data, np.random.choice(len(all_data), 1000, replace=False))</p>
<p>train_data=800<br>
test_data=200<br>
train_data, test_data = random_split(dataset_subset, [train_data, test_data])<br>
train_data.dataset.transform = image_transforms[‘train’]</p>
<p>train_loader = DataLoader(train_data, batch_size=batch_size, shuffle=True)</p>
<p>trainiter = iter(train_loader)<br>
features, labels = next(trainiter)<br>
print(features.shape, labels.shape)</p>
<hr>
<p>TypeError: default_collate: batch must contain tensors, numpy arrays, numbers, dicts or lists; found &lt;class ‘PIL.Image.Image’&gt;</p>

username: ptrblck
cooked: <p>Try to pass the transformation directly to the <code>ImageFolder</code> instead of trying to manipulate the internal attribute afterwards.</p>