Creating 3D Dataset/DataLoader with patches

banikr: <p>I have 20 3D nifty images which sizes are 172x220x156. I want to create a Dataset class and then a DataLoader made of patches of size 32x32x32 cropped from the images.<br>
Each image will have 500 patches like that. so the total number of patches should be 18x500. I have worked with the 2D silces like that before(please see the codes below I used for 2D)<br>
Is there any way to create similar Dataset class with patches from the 18 images??</p>
<pre><code class="lang-auto">class Dataset(data.Dataset):
    'characterizes a dataset for pytorch'
    def __init__(self, dir_data, list_IDs):
        self.dir_data = dir_data
        self.list_IDs = list_IDs

    def __len__(self):
        'denotes the total number of samples'
        return len(self.list_IDs)

    def __getitem__(self, item):
        'Generates one sample of data'
        #select sample
        path_mr = os.path.join(self.dir_data, 'mr', self.list_IDs[item])
        path_ct = os.path.join(self.dir_data, 'ct', self.list_IDs[item])

        X = Image.open(path_mr)
        X = np.array(X)  #np.shape(X) = (256,256)
        X = X[:,:,np.newaxis]  #np.shape(X) = (256,256,1)
        X = ToTensor()(X)
        y = Image.open(path_ct)
        y = np.array(y)[:,:,np.newaxis]
        y = ToTensor()(y)
        return X, y
</code></pre>

ptrblck: <p>How would you like to create these patches?<br>
If you would like to create them in a non-overlapping way, you would end up with<br>
<code>(172//32) * (220//32) * (156//32) = 120</code> patches.<br>
Could you explain, how these 500 patches should be created.</p>
<p>For the non-overlapping case you could load the image, use <code>unfold</code> to create the patches and return them in your <code>__getitem__</code>:</p>
<pre><code class="lang-python">x = torch.randn(172, 220, 156)
patches = x.unfold(2, 32, 32).unfold(1, 32, 32).unfold(0, 32, 32)
patches = patches.contiguous().view(-1, 32, 32, 32)
print(patches.shape)
&gt; torch.Size([120, 32, 32, 32])
</code></pre>
<p>The code you’ve provided for the 2D case doesn’t seem to create patches or am I missing something?</p>

banikr: <p>so I have two modality paired images: MR and CT, each having 172x220x156 images of 20 subjects.<br>
I am creating mask of the 3D data(based on intensity thresholding). Then in random permutation I am taking 500 center points inside the mask(so that I can avoid taking patches from outside the brain). Then I am cropping 32x32x32 patches from each of the 500 center points in both CT and MR images. To mention, MR is input and CT is ground truth.</p>
<p>Yes you are right, the codes I provided is for 2D network which I worked before by creating 156 2D image slices from 3D ones and that worked pretty well and I understood that as well too.<br>
But is there anyway I can do the same method with 3D patches now?</p>
<p>So far I have created two list(MR and CT) of 3D patches from 18 train subjects(2 are for test).</p>
<pre><code class="lang-auto">for i in filenames_train:
    path_mr = os.path.join(dir_train,'MR',i)
    MR = nib.load(path_mr)
    MR = np.array(MR.dataobj)
    path_ct = os.path.join(dir_train, 'CT', i)
    CT = nib.load(path_ct)
    CT = np.array(CT.dataobj)
    path_mask = os.path.join(dir_train, 'masks', i)
    mask = nib.load(path_mask)
    mask = np.array(mask.dataobj)

    X, y = get_paired_patch_3D(MR,CT,mask,num_centers=500,patchsize=32)

    patches_MR.append(X)
    patches_CT.append(y)
</code></pre>
<p><code>print(np.shape(patches_MR)) #(18, 500, 32, 32, 32)</code><br>
<code>print(np.shape(patches_CT)) #(18, 500, 32, 32, 32)</code></p>
<p>I will be using k-fold cross-validation and 3D UNet to synthesize CT images from MR.</p>
<p>Thanks for reply. Please let me know to clarify more if required.</p>

ptrblck: <p>Thanks for the information!<br>
It looks like your <code>get_paired_patch_3D</code> method already provides the patches from the MR and CT images.</p>
<p>If I understand your question correctly, you would now like to create a <code>Dataset</code> to yield a MR-CT pair as a single sample?</p>
<p>If that’s the case, you could simply view the patches as <code>patches_MR = patches_MR.reshape(-1, 32, 32, 32)</code> (same for <code>patches_CT</code>) and pass them to a custom <code>Dataset</code>:</p>
<pre><code class="lang-python">class MyDataset(Dataset):
    def __init__(self, patches_MR, patches_CT):
        self.patches_MR = patches_MR
        self.patches_CT = patches_CT
        
    def __getitem__(self, index):
        x = self.patches_MR[index]
        y = self.patches_CT[index]
        # Unsqueeze channel dimension
        x = x.unsqueeze(0)
        y = y.unsqueeze(0)
        return x, y
    
    def __len__(self):
        return len(self.patches_MR)
</code></pre>
<p>I assume your method to extract the patches keeps returns the corresponding pairs without any shuffling.</p>
<p>Let me know, if I misunderstood the question or use case.</p>

banikr: <p><strong>1.</strong> Yes <code>get_paired_patch_3D</code> function provides same location patches from the MR and CT images.</p>
<p><strong>2.</strong> I am not sure about the <strong>single sample</strong> thing. are you refering to the batch size?</p>
<p><strong>3.</strong> I tried things like this:</p>
<pre><code class="lang-auto"># print(np.shape(patches_MR))  #(18, 500, 32, 32, 32)
# print(np.shape(patches_CT))  #(18, 500, 32, 32, 32)

class Dataset(data.Dataset):
    'characterizes a dataset for pytorch'
    def __init__(self, patches_MR, patches_CT):
        self.patches_MR = patches_MR
        self.patches_CT = patches_CT
        # self.patches_MR = patches_MR
        # self.patches_CT = patches_CT

    def __len__(self):
        'denotes the total number of samples'
        return len(self.patches_MR)

    def __getitem__(self, index):
        'Generates one sample of data'
        #select sample
        x = self.patches_MR[index]
        y = self.patches_CT[index]
        # Unsqueeze channel dimension
        x = x.unsqueeze(0)
        y = y.unsqueeze(0)
        return x, y
</code></pre>
<p>and then in the train.py:</p>
<pre><code class="lang-auto">train_dataset = Dataset(patches_MR,patches_CT)
train_loader = data.DataLoader(train_dataset,batch_size=5,shuffle=True)
print('train directory has {} samples'.format(len(train_dataset)))
# np.shape(train_dataset)
# Out[242]: (18, 2, 500, 32, 32, 32)
</code></pre>
<p>As you can see the <code>train_dataset</code> has 2 channels: if I am not wrong one for MR and one for CT and did you use <code>unsqueeze</code> for that reason?</p>
<p><strong>4.</strong>  Yes while extracting patches I did not use any <code>shuffle</code> which I intend to use in the <code>torch.utils.data.DataLoader</code> as you can see in the codes.</p>
<p><strong>5.</strong> I want to feed the input(MR) and ground truth(CT) in the UNet/GAN as:</p>
<pre><code class="lang-auto">        for i, sample in enumerate(train_loader):
            netG.train()
            netD.train()
            time_batch_load = time.time() - time_batch_start
            time_compute_start = time.time()
            mr = sample[0].float().to(device) 
            ct = sample[1].float().to(device)
            batch_size_temp = len(sample[0])
            outputG = netG(mr)
</code></pre>

banikr: <p>I would like to add the following:</p>
<pre><code class="lang-auto">for i,sample in enumerate(train_loader):
    mr = sample[0].float().to(device)
    print('input sample shape of train_loader: {}'.format(mr.shape))
    break
</code></pre>
<p>gives :</p>
<pre><code class="lang-auto">input sample shape of train_loader: torch.Size([5, 1, 500, 32, 32, 32])

Process finished with exit code 0
</code></pre>
<p>which is not the exact thing I was expecting. I wanted 5/10 patches at a time. Based on the batch_size = 5 or 10.<br>
Do you think <code>transpose</code> function may come handy?</p>

ptrblck: <p>It seems you didn’t applied the suggested <code>reshape</code> operation on the data.<br>
Here is a sample code:</p>
<pre data-code-wrap="python"><code class="lang-python">
patches_MR = torch.from_numpy(
    np.random.randn(18, 500, 32, 32, 32).reshape(-1, 32, 32, 32))
patches_CT = torch.from_numpy(
    np.random.randn(18, 500, 32, 32, 32).reshape(-1, 32, 32, 32))


train_dataset = Dataset(patches_MR,patches_CT)
print('train directory has {} samples'.format(len(train_dataset)))
&gt; train directory has 9000 samples

train_loader = DataLoader(train_dataset,batch_size=5,shuffle=True)
x, y = next(iter(train_loader))
print(x.shape)
&gt; torch.Size([5, 1, 32, 32, 32])
print(y.shape)
&gt; torch.Size([5, 1, 32, 32, 32])
</code></pre>
<p>I think this is what you want, i.e. given <code>batch_size=5</code>, you’ll get 5 patches from each dataset for each iteration.</p>
<p>I’m wondering, why this seems to work:</p>
<aside class="quote no-group" data-username="banikr" data-post="5" data-topic="50861">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/banikr/48/54353_2.png" class="avatar"> banikr:</div>
<blockquote>
<pre data-code-wrap="python"><code class="lang-python"># np.shape(train_dataset) 
# Out[242]: (18, 2, 500, 32, 32, 32)
</code></pre>
</blockquote>
</aside>
<p><code>np.shape</code> throws an error if I pass my custom <code>Dataset</code>, which I would expect, but apparently you got some shape information?</p>

banikr: <p>Hi,<br>
<strong>1.</strong> yes, I haven’t used the <code>reshape</code> operation at that time.<br>
<strong>2.</strong> I am curious why have you included <code>np.random.randn</code> ?<br>
The CT and MR patch centers come from the exact same location from two images. I mean the patches must be <strong>paired</strong>.<br>
Won’t <code>np.random.randn</code> change the correspondence of patches in CT and MR based on <code>random.seed()</code>?<br>
<strong>3.</strong> I have used <code>reshape</code> in the following way:</p>
<pre><code class="lang-auto">patches_MR = np.array(patches_MR)
patches_MR = patches_MR.reshape(-1,32,32,32) #todo: which formats
patches_CT = np.array(patches_CT)
patches_CT = patches_CT.reshape(-1,32,32,32)
# patches_MR = np.array(patches_MR) #type(patches_MR) --&gt;  numpy.ndarray

train_dataset = Dataset(patches_MR,patches_CT)
train_loader = data.DataLoader(train_dataset,batch_size=5,shuffle=True)
print('train directory has {} samples'.format(len(train_dataset)))
# train directory has 9000 samples


for i,sample in enumerate(train_loader):
    mr = sample[0].float().to(device) 
    print('input sample shape of train_loader: {}'.format(mr.shape))
    #input sample shape of train_loader: torch.Size([5, 1, 32, 32, 32])
#    input sample shape of train_loader: torch.Size([5, 1, 32, 32, 32])
#    input sample shape of train_loader: torch.Size([5, 1, 32, 32, 32])
.
.
.

</code></pre>
<p>Now it works as you have suggested I guess.</p>
<p><strong>4.</strong> Its funny coz I also don’t know how it worked at that time:</p>
<pre><code class="lang-auto"># np.shape(train_dataset) 
# Out[242]: (18, 2, 500, 32, 32, 32)
</code></pre>
<p>but later it showed errors and I have corrected it as you have mentioned.</p>
<p>I have the following question:</p>
<p><strong>5.</strong>  Is it necessary to use the <code>class Dataset</code> and <code>torch.utils.data.DataLoader</code> to input the data to the network??<br>
I have seen PyTorch codes that do not use these two either.</p>

ptrblck: <ul>
<li>
<p>I just created some dummy data for the example. You should of course use your paired inputs, not the garbage data. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>
</li>
<li>
<p>It’s probably not really necessary, but wrapping the data in a <code>Dataset</code> allows you e.g. to add some transformations later. Using a <code>DataLoader</code> on the other hand allows to create batches easily, shuffle the data (the pairs will still be valid), use multiple workers etc. It’s just a clean approach in my opinion, but since you already have the data in memory, you could also just index it manually.</p>
</li>
</ul>

banikr: <p>Haha, I should’ve guessed that.<br>
I will run the experiments in some network and let’s hope that it works.</p>
<p>Thanks, appreciate it.</p>

Aliktk: <p>hello  banikr,<br>
I have question I read your post and all the conversation. Actually I am making data loader for MRI images collected from ADNI. I loaded a single image from training folder now I want to load all the MRI images in a iteration way and than apply some neural network for classification purposes.<br>
Please help me that how you load your whole MRI data from the directory<br>
I have 900 MRI images in three different folder i.e Alzheimer have three main classes<br>
CN, MCI, AD so I want to load all the data from each folder but how I to do?<br>
Further more I read 1000 post and tutorial but I couldn’t get an idea to implement as I am not much expert  in pytorch and 3D data handling.<br>
I am using following IDE and libraires<br>
IDE- Spyder<br>
using Pytorch and tensorflow<br>
python 3.7<br>
Thanks in advance</p>

banikr: <p>Hey,<br>
I did not load the whole MRI image to the data loader. The MR images I am using are of size 172x220x156 so it will exceed the memory Cuda cores can load.<br>
For image synthesis, I created patches of 10000 per image and augmented the data. In your case of classification, it should be similar.<br>
Then I am using regression analysis/prediction from MR image which will not work in patch-based training…so I subsampled the image to reduce the number of voxel size per image.<br>
Then the PyTorch data loader should work fine.<br>
Let me know if you need more help.<br>
I would suggest you use Jupyter notebook or Pycharm IDE for coding. I find them easy to use and feasible. Use python 3.6 if possible, not all the libraries support 3.7 yet.<br>
Since it is Pytorch help forum I would ask you to stick to it, eh… <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

Bdtask: <p>How to make use of the  <code>torch.utils.data.Dataset</code>  and  <code>torch.utils.data.DataLoader</code>  on your own data (not just the  <code>torchvision.datasets</code> )?</p>
<p>Is there a way to use the inbuilt  <code>DataLoaders</code>  which they use on  <code>TorchVisionDatasets</code>  to be used on any dataset?</p>

ptrblck: <p>Yes, that’s possible and you can write your own <code>Dataset</code> implementation and just pass it to a <code>DataLoader</code>.<br>
Have a look at <a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html">this tutorial</a> for an example.</p>

Aliktk: <p>Thanks <strong>banikr</strong> for your valuable reply<br>
that my objective that I passed the whole MR image into my network and my network just classify in their respective classes. But now i knows about that no any method which take directly a 3D image as input file and than some processing by CNN or whatever the network is and than classify into their classes, every one use patch wise input into their network. Furthermore, in my project I used all the ADNI data for that I don’t use augmentation but directly processed all my MR images.<br>
yes you’re right that regression analysis will not help in this regard, you have to use neural network for that purpose as suggestion<br>
will now I am very use to in spyder IDE as its on top of most using IDE now days.<br>
if you don’t have issue could please show some of code snippet which from dataloading for guidance.<br>
Thanks for reading such a long reply:innocent:<img src="https://discuss.pytorch.org/images/emoji/apple/innocent.png?v=9" title=":innocent:" class="emoji" alt=":innocent:"></p>

Aliktk: <p>Hi  <strong>baniker</strong> How to convert a single or bunch of MRI image having <strong>.nii</strong> format into patches?<br>
also please guide me how to subsample same image using pytorch?</p>

Elahe_Heydari: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a><br>
I have a question about unfold. I want to extract patches from my dataset. I use medicaltorch libarary to loading data. If i use unfold , It has error, I think when I load data by using dataloader, it doesn’t access to data. what can I do?<br>
Thanks.</p>
<p>ROOT_DIR= “/home/elahe/data/dataset/”<br>
img_list = os.listdir(os.path.join(ROOT_DIR,‘trainnii’))<br>
label_list = os.listdir(os.path.join(ROOT_DIR,‘labelsnii’))<br>
print(img_list[1])<br>
img_list= (i.unfold(2, 32, 32).unfold(1, 32, 32).unfold(0, 32, 32) for i in img_list)<br>
label_list = (i.unfold(2, 32, 32).unfold(1, 32, 32).unfold(0, 32, 32) for i in label_list)</p>
<p>filename_pairs = [(os.path.join(ROOT_DIR,‘trainnii’,x),os.path.join(ROOT_DIR,‘labelsnii’,y)) for x,y in zip(img_list,label_list)]<br>
print(filename_pairs)<br>
train_transform = transforms.Compose([<br>
mt_transforms.Resample(0.25, 0.25),<br>
mt_transforms.ElasticTransform(alpha_range=(40.0, 60.0),<br>
sigma_range=(2.5, 4.0),<br>
p=0.3),<br>
mt_transforms.ToTensor()]<br>
)<br>
train_dataset = mt_datasets.MRI2DSegmentationDataset(filename_pairs,transform=train_transform)<br>
dataloader = DataLoader(train_dataset, batch_size=2,collate_fn=mt_datasets.mt_collate)</p>

ptrblck: <p>What error do you get?<br>
<code>unfold</code> is a method which should be called by or on a tensor. Based on your code snippet it looks like you are calling it on a file path (string).<br>
Load the images, transform them to tensors, and then call <code>unfold</code> on them.</p>

Elahe_Heydari: <p>I apply it after transform, It worked. Thanks a lot.<br>
I have another question, Is there any way for pairing label and without label images in 3D, like “MRI2DSegmentationDataset”?<br>
“MRI2DSegmentationDataset” pair images in 2D, but I want to pair in 3D.<br>
Or should  i use the patches and transform them to 2D?<br>
can I train my data by using patch in 3D?</p>

ptrblck: <p>I’m not sure what “pairing” means in this context.<br>
If you want to work on a segmentation use case for 3D data, it should work in the same manner as for 2D data (just with an additional dimension).</p>