How to load all the .nii from the directory without augmentation using pytorch dataloader?

username: Aliktk
cooked: <p>Actually I am making data loader for MRI images collected from ADNI. I loaded a single image from training folder now I want to load all the MRI images as it is,  in a iterative way and than apply some neural network for classification purposes.<br>
Please help me that how you load your whole MRI data from the directory<br>
I have 900 MRI images in three different folder i.e Alzheimer have three main classes<br>
CN, MCI, AD so I want to load all the data from each folder but how I to do?<br>
Further more I read 1000 post and tutorial but I couldn’t get an idea to implement as I am not much expert  in pytorch and 3D data handling.<br>
I am using following IDE and libraries<br>
IDE- Spyder<br>
using Pytorch and tensorflow<br>
python 3.7<br>
Thanks in advance</p>

username: mailcorahul
cooked: <p>Have you checked the below tutorial on data loading?<br>
<a href="https://pytorch.org/tutorials/beginner/data_loading_tutorial.html" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/data_loading_tutorial.html</a></p>

username: Aliktk
cooked: <p>thanks for your reply mailcorahul<br>
ye i checked already but couldn’t get idea how to load my MRI .nii file, but here I am talking about 3D and 4D data which is related to medical image analysis. I am stuck with dataloading step as I have 17GB of data which is in the form of nii file not any simple image file. I want to load them all in iterative way into my memory and after that give as input for my CNN.</p>

username: czotti
cooked: <p>You will need a custom <a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.Dataset" rel="nofollow noopener">Dataset</a>.</p>
<p>How I use the custom for my use case (I assume you use nibabel to handle nifti files), I preload all the file handle with <a href="https://nipy.org/nibabel/reference/nibabel.loadsave.html?highlight=load#load" rel="nofollow noopener">nibabel</a>.<br>
During the iteration you could get the data with the <code>np.asarray(nii_image.dataobj)</code>. If you use <code>get_data()</code> you should specify the caching by default it keeps every loaded image in cache this could overload you system memory by loading to many image.</p>
<pre><code class="lang-auto">class MyCustomDataset(Dataset):
    def __init__(self, path):
        # load all nii handle in a list
        self.images_list = [nib.load(image_path) for image_path in path]

    def __len__(self):
        return len(self.images_list)

    def __getitem__(self, idx):
        nii_image = self.images_list[idx]
        data = torch.from_numpy(np.asarray(nii_image.dataobj))
        # find how to retrieve the target
        return data, target
</code></pre>
<p>You will need a ways to associate the target of your <code>__getitem__</code>  function, this part depend on how you will handle the data/directory, etc.</p>

username: Aliktk
cooked: <p>thanks <strong>czotti</strong> for quick reply<br>
Yes dude i check the mentioned links and also create the same code which is given in the link and you provided here below is</p>
<pre><code class="lang-auto">class Dataloder_img(data.Dataset):
    def __init__(self,root_dir,transforms ):
        self.root_dir = root_dir
        self.seg_dir = seg_dir
        self.transforms = transforms
        self.files = os.listdir(self.root_dir)
        self.lables = os.listdir(self.seg_dir)
        print(self.files)
    
    def __len__(self):
        return len(self.files)
    
    def __getitem__(self,idx):
        img_name = self.files[idx]
        label_name = self.lables[idx]
        img = nib.load(os.path.join(self.root_dir,img_name)) 
        #change to numpy
        img = np.array(img.dataobj)
        #change to PIL 
#        img = Image.fromarray(img.astype('uint8'), 'RGB')    
        print(img.size)
        
        label = nib.load(os.path.join(self.seg_dir,label_name))
        #change to numpy
        label = np.array(label.dataobj)
        #change to PIL 
#        label = Image.fromarray(label.astype('uint8'), 'RGB')
        
        print(label.size)
        
        if self.transforms:
            img = self.transforms(img)
            label = self.transforms(label)
            return img,label
        else:
            return img, label
full_dataset = Dataloder_img('C:/Users/Ali ktk/.spyder-py3/dataloader/data/train/1', 'C:/Users/Ali ktk/.spyder-py3/dataloader/data/train/1/ADNI_136_S_0300_MR_MPR__GradWarp__B1_Correction__N3__Scaled_Br_20080529142830882_S50401_I107759.nii' ,tfms.Compose([tfms.RandomRotation(180).tfms.ToTensor()]))
                                   

train_size = int(0.8 * len(full_dataset))
val_size = len(full_dataset) - train_size
train_dataset, val_dataset = torch.utils.data.random_split(full_dataset, [train_size, val_size])
train_loader = data.DataLoader(train_dataset,shuffle=False,batch_size=bs)
val_loader = data.DataLoader(val_dataset,shuffle=False,batch_size=bs)

test_img, test_lb = next(iter(full_dataset))
print(test_img[0].shape)
plt.imshow(test_img[1])
plt.show()
</code></pre>
<p>But its doesn’t work for me as<br>
the error is shown below</p>
<pre><code class="lang-auto">full_dataset = Dataloder_img('C:/Users/Ali ktk/.spyder-py3/dataloader/data/train/1', 'C:/Users/Ali ktk/.spyder-py3/dataloader/data/train/1/ADNI_136_S_0300_MR_MPR__GradWarp__B1_Correction__N3__Scaled_Br_20080529142830882_S50401_I107759.nii' ,tfms.Compose([tfms.RandomRotation(180).tfms.ToTensor()]))

AttributeError: 'RandomRotation' object has no attribute 'tfms'
</code></pre>
<p>Here when I remove this error than nothing is working except error.<br>
Please help in this regard that how I can get my all the data which are in my directory.<br>
Thanks in advance</p>

username: Aliktk
cooked: <p>I tried this method without using dataset class but still its have error. Actually I have not much expertise in coding side but I trying to be a good programmer. Here is the code for another method which I am using.</p>
<pre><code class="lang-auto">cwd = 'C:/Users/Ali ktk/.spyder-py3/dataloader/data/train/1'
#data_dir = 'C:/Users/Ali ktk/.spyder-py3/dataloader/data/test/0/ADNI_137_S_0800_MR_MPR__GradWarp__N3__Scaled_Br_20081024125416989_S56178_I123506.nii'


#cwd = os.getcwd()#return current working directory
print(cwd)
#debugger
patient_id = os.listdir('C:/Users/Ali ktk/.spyder-py3/dataloader/data/train/1') #create a list that encompasses all the images we downloaded
len(patient_id)
print(patient_id)
#

#filename = glob.glob('C:/Users/Ali ktk/.spyder-py3/dataloader/data/train/1/ADNI_137_S_0800_MR_MPR__GradWarp__N3__Scaled_Br_20081024125416989_S56178_I123506.nii' + 'C:/Users/Ali ktk/.spyder-py3/dataloader/data/train/1/ADNI_137_S_0800_MR_MPR__GradWarp__N3__Scaled_Br_20081024125416989_S56178_I123506.nii')
#print(filename)    

dataimage = [patient_id]#[patient_id, image_matrix_normalized]
for patient in patient_id:
    for root, dirs, files in os.walk(cwd + "/ADNI/" + patient):
        flag = 0 
        for file in files:
            if file.endswith(".nii"):
                print(os.path.join(root, file))
                if flag &lt; 1:
                    datapath.append(os.path.join(root, file))
                    print(os.path.join(root, file))
                    print(flag)
                    flag = flag + 1
datapath = [patient_id, 'C:/Users/Ali ktk/.spyder-py3/dataloader/data/train/1'] #[patient_id, path]
tar_dim = [256, 256, 160]

for path in datapath:
    mri = nib.load(cwd).get_data()
    mri = (mri - mri.min())/(mri.max() - mri.min())
    mri = mri - mri.mean()
    tar_mri = ndimage.zoom(mri,[tar_dim[0]/mri.shape[0],tar_dim[1]/mri.shape[1], tar_dim[2]/mri.shape[2]], order = 1)

    print(mri)
    print(mri.mean())
    print(mri.var())
    print(tar_mri.shape)
    plt.imshow(mri[125,:,:],'gray')
    plt.show()
    
    dataimage.append(tar_mri)
    
</code></pre>
<p>after running this code I got below error which belongs to unsupported files</p>
<pre><code class="lang-auto">ImageFileError: Cannot work out file type of "C:/Users/Ali ktk/.spyder-py3/dataloader/data/train/1"
</code></pre>
<p>Please any one who already work with Nii file format help me here?</p>

username: czotti
cooked: <p>For the first error:</p>
<blockquote>
<p>AttributeError: ‘RandomRotation’ object has no attribute ‘tfms’</p>
</blockquote>
<p>It’s a typo <code>.</code> → <code>, </code>:</p>
<pre><code class="lang-auto">tfms.Compose([tfms.RandomRotation(180), tfms.ToTensor()])
</code></pre>
<p>Also your <code>__init__</code> functione have 2 parameters (<code>self</code> is used by python and represent the object instance) <code>root_dir</code> and <code>transforms</code>.<br>
On the line <code>full_dataset = Dataloader_img(...)</code> you provides 3 parameters, the directory, the nii file and transform object. At least you need to removes the nii file.</p>
<p>For your second error it’s normal you tried to use a directory as input for <code>nib.load(cwd)</code> replace this with a <code>.nii</code> file and you should be fine.</p>

username: Aliktk
cooked: <aside class="quote no-group" data-username="czotti" data-post="7" data-topic="60938">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/czotti/48/8031_2.png" class="avatar"> czotti:</div>
<blockquote>
<p>tfms.Compose([tfms.RandomRotation(180), tfms.ToTensor()])</p>
</blockquote>
</aside>
<p>still its not working I am talking about the 2nd pecide of code.<br>
as it has generated the error of not handling this type of file from its built in files.<br>
have a look on this screenshot. I change the mentioned and replace with nii file but it show below error<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/5/c5b937e9f4ec196c1da388de9d3819a3a8abcf3b.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/c5b937e9f4ec196c1da388de9d3819a3a8abcf3b" title="Screenshot_7"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/5/c5b937e9f4ec196c1da388de9d3819a3a8abcf3b_2_690x316.jpeg" alt="Screenshot_7" data-base62-sha1="sd8Tcjjf6XhCLZH40uAAiGYfWIz" width="690" height="316" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/5/c5b937e9f4ec196c1da388de9d3819a3a8abcf3b_2_690x316.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/c/5/c5b937e9f4ec196c1da388de9d3819a3a8abcf3b_2_1035x474.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/c/5/c5b937e9f4ec196c1da388de9d3819a3a8abcf3b.jpeg 2x" data-dominant-color="F3F3F0"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot_7</span><span class="informations">1327×608 176 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: czotti
cooked: <p>As the error reports it didn’t load the <code>.nii</code> file but still the <code>C:\Users\....\train\1</code> directory.</p>
<p>In your exemple you should changes these lines:</p>
<pre><code class="lang-auto">datapath = [patient_id, 'C:/Users/Ali ktk/.spyder-py3/dataloader/data/train/1'] #[patient_id, path]
</code></pre>
<p>into</p>
<pre><code class="lang-auto">datapath = patient_id
</code></pre>
<p>And to load the images properly with nibabel change</p>
<pre><code class="lang-auto">mri = nib.load(cwd).get_data()
</code></pre>
<p>into</p>
<pre><code class="lang-auto">mri = nib.load(path).get_data()
</code></pre>
<p>BTW all these questions are not related to pytorch at all maybe it’s more suited to go on the <code>nibabel</code> forum/gitter/whatever they use to communicate, to learn how to properly load medical imaging data</p>

username: Aliktk
cooked: <p>Thanks <strong>Czotti</strong> I am trying and you are right that these issue related with <code>nibable</code><br>
forum,<br>
actually I want to discuss it here because millions of user are now turning to pytorch. If they have experience with medical imaging than they will solve my problem here more in suitable way<br>
thanks for your valuable time</p>

username: Aliktk
cooked: <p>I am still stuck in the same problem.<br>
Let me know that<br>
is I load MRI 3D image having 256,256,166 dimension directly without patching tham or resampling tham.<br>
My target in to load all of tham without any pre-processing steps or patching to the input.<br>
Is it possible&gt;&gt;&gt;???<br>
furthermore If I am wrong please mentioned the steps and guide me please to get out from here.<br>
Thanks in advance<br>
<strong>Clement Zotti</strong> and <strong>Raghul Asokan</strong></p>