Pytorch customized data loader

Daniel_Joseph: <p>Hello everyone, I have just started learning Pytorch and i got a problem while trying to create dataloader from my customized datasets  which is contains 20 files “<strong>data_x.mat</strong>” stored in specific folder, I need to use them with data-loader, can anyone help me how to create this class to get iterable batches :<br>
Mydata is</p>
<ul>
<li>10 files each is dict {“Training_Patches”: shape(760,120,21,21)  /  “Label”: shape(1,760)}  stored in .mat file</li>
<li>10 files each is dict {“Testing_Patches”: shape(420,120,21,21)  /  “Label”: shape(1,420)}  stored in .mat file<br>
Here is what i thing it should looks like, Any suggestion,ideas, help would be Appreciated</li>
</ul>
<pre><code class="lang-auto">class Datasets(Dataset):
    def __init__(self):
        self.tensors = []
        self.labels = []
       for i  in range(len(data_i["Training_patches"])):
                    self.tensors.append(data_i["Training_patches"])
                    self.labels.append(data_i["labels"])

    def __getitem__(self, index):
        # return one item on the index
        return

    def __len__(self):
        # return the data length
        return 

</code></pre>

ptrblck: <p>Would you like to load each file as one sample or is a whole batch saved in each file?<br>
Were you able to load the data using <code>scipy.io.loadmat</code>?<br>
Here is a dummy example for your dataset:</p>
<pre><code class="lang-python">
class MyDataset(Dataset):
    def __init__(self, mat_paths):
        self.paths = mat_paths
        
    def __getitem__(self, index):
        # Load .mat
        data = io.loadmat(self.paths[index])
        x = torch.from_numpy(data['Training_patches'])
        y = torch.from_numpy(data['labels'])
        
        return x, y
    
    def __len__(self):
        return len(self.paths)
</code></pre>

Daniel_Joseph: <p>Thanks ptrblck for replying , what i need is to load the whole files in the directory as one sampler to use it with the torch.Dataloader as follow :</p>
<pre><code class="lang-auto">Train_dst = MyDataset(path=path, train=True)
Test_dst = MyDataset(path=path, train=False)

TrainLoader = torch.utils.data.DataLoader(dataset=Train_dst, batch_size=50, shuffle=True )
TestLoader = torch.utils.data.DataLoader(dataset=Test_dst, batch_size=50, shuffle=True )

</code></pre>

ptrblck: <p>I’m not sure I understand your use case properly.<br>
If each file should be a sample, you could only use a batch size of 10, as you only have 10 files.<br>
Did my code sample work for you?</p>

keyu_Chen: <p>I have a Image.mat file with 2656 image samples.<br>
It has 4 key elements, “img” key contains the data with size(2656, 4097), the last column is the class label.<br>
How can I build a dataset based on this .mat file?</p>

ptrblck: <p>I assume the data fits completely into memory?<br>
If so, you could load the <code>.mat</code> file using <code>scipy</code> in your <code>Dataset</code>'s <code>__init__</code> method, get a single sample in <code>__getitem__</code> by indexing and returning the tensor via <code>torch.from_numpy</code>.</p>

saba: <p>Hi Ptrblck,</p>
<p>I wrote my custom data loader based on your guidance. The code is correct and work. The problem is that by changing the BatchSize, It does not work properly, every time it gives me data in 64 Batchsize regardless of the Batchsize number. Would you please help me with that?</p>
<pre><code class="lang-auto">workers=2
Batchsize=128

datarootNorm='//homeHealthytorchpatch_v1/'

datasetNorm=DatasetLoad(datarootNorm,transforms=transform,debug=False,ii=1)

dataloaderNormal = torch.utils.data.DataLoader(datasetNorm, batch_size=Batchsize,shuffle=True, num_workers=workers)


class DatasetLoad():

    def __init__(self, root_dirTrain,transforms, debug,ii):
        self.patches, self.labels = None, None
        self.ii=ii
        patch_path = os.path.join(root_dirTrain,'Healthyzone'+str(self.ii)+".csv")
        ## 15000x1x64x64
        self.patches=torch.load(patch_path) 
        # self.patches = scipy.io.loadmat(patch_path)['ImagesNormal']
        # patch_path = os.path.join(root_dirTrain,"NegTraining11_v1"+".mat")
        # self.patches = scipy.io.loadmat(patch_path)['NegTraining11_v1']
        self.transforms = transforms
        self.debug = debug

    def __getitem__(self, index):

        patchF = self.patches[index,:, : ,:]

        if self.transforms is not None:

            patchF = self.transforms(patchF)

        return patchF


    def __len__(self):
        return self.patches.shape[-1]

</code></pre>

ptrblck: <p>What shape does <code>self.patches</code> have? Is it <code>[15000, 1, 64, 64]</code>?<br>
If so, I assume <code>dim0==15000</code> are the number of samples?<br>
If that’s also correct, I think you are returning the wrong <code>__len__</code>, as <code>self.patches.shape[-1]</code> would return 64 instead of 15000, so your max. batch size would be 64 and you would only load <code>patches[:64]</code>.</p>

saba: <p>Many thanks let me check</p>

saba: <p>yes you are right I correct it. I really appreciate.</p>

saba: <p>Hi Ptrblck,</p>
<p>I am using spyder editor.<br>
My code was working but now when I want to use data loader and get the data from that it gives me this error"<br>
[SpyderKernelApp] WARNING | WARNING: attempted to send message from fork<br>
{‘header’: {‘version’: ‘5.3’, ‘date’: datetime.datetime(2021, 3, 29, 6, 7, 4, 493062, tzinfo=datetime.timezone.utc), ‘session’: ‘5b99e49d-1c858153ee7c8a8ade190ea5’, ‘username’: ‘mom008’, ‘msg_type’: ‘comm_msg’, ‘msg_id’: ‘5b99e49d-1c858153ee7c8a8ade190ea5_84’}, ‘msg_id’: ‘5b99e49d-1c858153ee7c8a8ade190ea5_84’, ‘msg_type’: ‘comm_msg’, ‘parent_header’:"<br>
The code is</p>
<pre><code class="lang-auto">for ii in range(1,7,1):
    
    datarootAbNorm= '//home/mom008//work/RoiClassification/radius_32_32/selectedROIs/zone_'+str(ii)+'/'
    
    datasetAbNorm=CustomDataSetZoneAbnormal(datarootAbNorm,transforms=transform,zone=ii)

    dataloaderAbnormal = torch.utils.data.DataLoader(datasetAbNorm, batch_size=1,shuffle=True, num_workers=workers)
    
    for i, data1 in enumerate(dataloaderAbnormal, 0):
</code></pre>

ptrblck: <p>Interactive interpreters (such as the one used in Spyder) don’t work well with multiprocessing, so you could use single workers while (live) developing in Spyder and add multiprocessing later (and execute the script from the terminal or as a standalone script in the IDE).</p>

saba: <p>I think the problem is for this part. I want to load the image which the name is in the defined list (not all). and with this command nothing for some times nothing is  passed.</p>
<pre><code class="lang-auto">class CustomDataSetZoneAbnormal_HMScan():
    def __init__(self, main_dir, dirusedList,transforms,zone):
        self.main_dir = main_dir
        self.zone=zone
        self.transforms = transforms
        self.dirusedList=np.load(dirusedList)

        all_imgs = os.listdir(main_dir)
        self.total_imgs = sorted(all_imgs)
       


    def __getitem__(self, idx):
                    
        cc=self.total_imgs[idx]
        ccSplit= cc.split('.')
        ccSplitsec=ccSplit[-2].split('_')
        ccSplitTh=ccSplitsec[:-2]
        for ii in range(len(self.dirusedList)):

            if ccSplitTh==self.dirusedList[ii]:

                img_loc = os.path.join(self.main_dir, self.total_imgs[idx])

                image11=mpimg.imread(img_loc).astype(float)

                return image11
           
            else:
                continue    
    
    def __len__(self):
        return len(self.total_imgs)
</code></pre>

hamidreza_hayatjou: <p>Hello everyone, I have a Image .mat file with 60556 image samples. It has 2 key elements, “image_data” and “image_labels”. When I execute the following command, the result is as follows :</p>
<pre><code class="lang-auto">data = io.loadmat('/dataset/Train48d48.mat')
image_data = data['image_data']
image_labels = np.squeeze(data['image_labels'])
print(image_data.shape, image_labels.shape)
</code></pre>
<p>(60556, 48, 48, 3) (60556,)<br>
i want build a dataloader. my code is  :</p>
<pre><code class="lang-auto">class MYDATASET(Dataset):
    def __init__(self, mat_paths):
        self.mat_paths ='/dataset/Train48d48.mat'
        
    def __getitem__(self, index):

        data = io.loadmat(self.mat_paths[index])
        x = data['image_data']
        y = np.squeeze(data['image_labels'])
        x = torch.from_numpy(x)
        y = torch.from_numpy(y)       
        return x, y
    
    def __len__(self):
        return len(self.mat_paths)
</code></pre>
<pre><code class="lang-auto">trainset = MYDATASET(mat_paths='/dataset')
trainloader = torch.utils.data.DataLoader(trainset, batch_size=128, shuffle=True, num_workers=2)
</code></pre>
<p>I was expecting after executing this command :</p>
<pre><code class="lang-auto">one_train_batch_imgs, one_train_batch_lbls = next(iter(trainloader))
print(one_train_batch_imgs.shape)
</code></pre>
<p>See the result below.<br>
torch.Size([128, 3, 48, 48])<br>
But I have an error like this : " FileNotFoundError: [Errno 2] No such file or directory: ‘d.mat’ "<br>
Where is my mistake? Any suggestion,ideas, help would be Appreciated.</p>

my3bikaht: <aside class="quote no-group" data-username="hamidreza_hayatjou" data-post="15" data-topic="30077">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hamidreza_hayatjou/48/45385_2.png" class="avatar"> hamidreza_hayatjou:</div>
<blockquote>
<p><code>'/dataset/Train48d48.mat'</code></p>
</blockquote>
</aside>
<p>this path shouldn’t work in first place. You are trying to load something from the root subdirectory (add . in front to point to current dir)</p>
<p>Another thing is:</p>
<aside class="quote no-group" data-username="hamidreza_hayatjou" data-post="15" data-topic="30077">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hamidreza_hayatjou/48/45385_2.png" class="avatar"> hamidreza_hayatjou:</div>
<blockquote>
<pre><code class="lang-auto">    def __getitem__(self, index):
        data = io.loadmat(self.mat_paths[index])
</code></pre>
</blockquote>
</aside>
<p>You are loading whole dataset every time single item is requested by the dataloader. Consider moving this part to <code>__init__</code>, otherwise it will bottleneck your loop.</p>

hamidreza_hayatjou: <p>Thanks for your attention. But unfortunately I did not get a result.</p>

my3bikaht: <aside class="quote no-group" data-username="hamidreza_hayatjou" data-post="15" data-topic="30077">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hamidreza_hayatjou/48/45385_2.png" class="avatar"> hamidreza_hayatjou:</div>
<blockquote>
<p><code>io.loadmat('/dataset/Train48d48.mat')</code></p>
</blockquote>
</aside>
<p>Ah, I see.<br>
First, I don’t think you can load specific element from *.mat file by providing [index]. You are loading whole dataset every time, that’s why I recommended to move loading part to <code>__init__</code> to do that only once.</p>
<p>Second, <code>self.mat_paths[index]</code> in <code>io.loadmat(self.mat_paths[index])</code> will be processed as a string and return you letter at position <code>index</code>. Since your path is <code>'/dataset/Train48d48.mat'</code>, dataloader requested item with index 1 or 17, which is letter <code>d</code>, then loadmat function appended <code>.mat</code> as file extension.</p>
<p>It should be something like this</p>
<pre><code class="lang-auto">class MYDATASET(Dataset):
    def __init__(self, mat_paths):
        self.data = io.loadmat(mat_paths)
    def __getitem__(self, index):
        x = self.data['image_data'][index]
        y = np.squeeze(self.data['image_labels'])[index]
        x = torch.from_numpy(x)
        y = torch.from_numpy(y)       
        return x, y
    def __len__(self):
        return len(self.data['image_data'])
</code></pre>

hamidreza_hayatjou: <p>thank you Sergey. You helped me a lot. I was able to achieve the result with a few changes. I will leave the code for those who may see it later.</p>
<pre><code class="lang-auto">class MYDATASET(data.Dataset):
    def __init__(self, mat_paths, transform=None):

        self.mat_paths = '/dataset/Train48d48.mat'
        self.data = io.loadmat(mat_paths)
        self.transform = transform

    def __getitem__(self, index):
        x = self.data['image_data'][index]
        y = np.squeeze(self.data['image_labels'])[index]

        if self.transform :
            x = self.transform(x)     
        return x, y

    def __len__(self):
        return len(self.data['image_data'])
</code></pre>
<pre><code class="lang-auto">transform = transforms.Compose([
    transforms.ToPILImage(),
    transforms.ToTensor()
])

trainset = MYDATASET(
    mat_paths='/dataset/Train48d48.mat', transform=transform)

trainloader = torch.utils.data.DataLoader( trainset, batch_size=128, shuffle=True)
</code></pre>