What is an appropriate usage of ImageFolder/DatasetFolder in this situation (images stored in a batch-ed manner)?

vaseline555: <p>Dear folks,</p>
<p>hello, I am a quite newbie in deep learning with PyTorch…</p>
<p>I am trying to build an image classification model using PyTorch,</p>
<p>but in the process of pre-processing dataset, I’ve stuck with a problem.</p>
<p>Problem)</p>
<ul>
<li>
<p>I want to read a bunch of <code>.npy</code> files in a directory (e.g. ‘/training’)</p>
</li>
<li>
<p>One of the <code>.npy</code> files consists of tensors with shape of <code>(N, C, H, W)</code>, i.e. mini-batch of images</p>
<p>(e.g. file1: <code>(1592, 3, 224, 224)</code>, file2: <code>(1100, 3, 224, 224)</code>, …)</p>
</li>
<li>
<p>The size of mini-batch is different from each other (1592, 1100, 683, …)</p>
</li>
</ul>
<p>Questions)</p>
<ul>
<li>
<p>AFAIK, <code>DatasetFolder</code> or <code>ImageFolder</code> module of <code>torchvision.datasets</code> are used for the directory having a list of single image files. (e.g. 1592 independent <code>.npy</code> files, of which shape is <code>(1, 3, 224, 224)</code>)</p>
</li>
<li>
<p>In my case, is there any recommended way to deal with the situation?<br>
(I finally want to use <code>DataLoader</code> after constructing a (custom) <code>DatasetFolder</code>/<code>ImageFolder</code> module)</p>
</li>
<li>
<p>I want to avoid the situation of reading each mini-batch <code>.npy</code> file, split them by a sample <code>(1, C, H, W)</code>, and save them again in a separate file…</p>
</li>
</ul>
<p>Thank you, in advance.</p>
<p>Have a good day!</p>

ptrblck: <p>There are different way of handling this use case.</p>
<p>The simplest would be to load the complete data into memory and just use some conditions on the index to select the right numpy array where the current sample is located to load it.</p>
<p>The other approach would be to create a separate <code>Dataset</code> for each numpy array.<br>
Based on the length of the underlying numpy arrays you could then create the indices and shuffle them (if shuffling is needed).</p>
<p>Once this is done you could then use <code>Subset</code> to pass these indices and enforce the shuffling, and pass all datasets into <code>ConcatDataset</code>.</p>
<p>Finally wrap the <code>ConcatDataset</code> into a <code>DataLoader</code> and it should make sure to load the data sequentially (but shuffle it inside each dataset).</p>
<p>Here is an example code snippet showing this approach, as the explanation might be a bit vague:</p>
<pre><code class="lang-python">from torch.utils.data import TensorDataset, DataLoader, Subset, ConcatDataset

dataset1 = TensorDataset(torch.arange(10).view(10, 1))
dataset2 = TensorDataset(torch.arange(10, 20).view(10, 1))

idx1 = torch.randperm(len(dataset1))
idx2 = torch.randperm(len(dataset1))

dataset1 = Subset(dataset1, idx1)
dataset2 = Subset(dataset2, idx2)

datasets = ConcatDataset((dataset1, dataset2))

loader = DataLoader(datasets, batch_size=5)
for data in loader:
    print(data)
</code></pre>
<p>Note that this approach would also load the complete dataset into memory.<br>
If you want to avoid it, you could create the “next” dataset once the old dataset was used completely.</p>
<p>Let me know, if any approach would work for you.</p>

vaseline555: <p>Sorry for super late reply.</p>
<p>Thanks to your insightful answer, I could deal with all problems I’ve faced!</p>
<p>For dealing with OOM problem, I’ve utilized <code>np.memmap</code> function for loading data directly from the storage.</p>
<p>Here is code snippet:</p>
<pre><code class="lang-auto">class PartialDataset(Dataset):
    """
    * Description: custom `Dataset` module for processing `.npy` files (N, C, H, W) (N &gt; 1) grouped by date
    - i.e. mini-batched .npy file stored by date
    - Therefore, the number of samples, 'N', is different from each other...
    """    
    def __init__(self, read_path, date, transform=None):
        """
        * Arguments:
        - read_path (string): path of `.npy` files
        - data (string): date(yymmdd) as a file name
        - transform (callable, optional): optional transform to be applied on a sample
        """
        self.transform = transform
        self.path = read_path
        self.date = date

        self.data = self.read_memmap(f'{os.path.join(self.path, self.date)}.npy')
        
    def read_memmap(self, file_name):
        """
        * Descripton: read `np.memmap` file from the directory
        
        * Argument:
        - file_name (string): path of '.npy' and '.npy.conf' files
        
        * Output:
        - whole data loaded in a memory-efficient manner (np.memmap) 
        """
        with open(file_name + '.conf', 'r') as file:
            memmap_configs = json.load(file) 
            return np.memmap(file_name, mode='r+', shape=tuple(memmap_configs['shape']), dtype=memmap_configs['dtype'])

    def __getitem__(self, index):
        """
        * Description: function for indexing samples
        
        * Argument:
        - index (int): index of the sample
        
        * Output:
        - input data, output data (torch.Tensor, torch.Tensor)
        - (batch_size, 4 (Mask(0 - background, 1 - foreground) / input1 / input2 / input3), height, width), (batch_size, output, height, width)
        """
        
        mask = torch.Tensor(self.data[index, 0, :, :]).reshape(1, PATCH_HEIGHT, PATCH_WIDTH)
        inputs = torch.Tensor(self.data[index, 2:4, :, :])
        output = torch.Tensor(self.data[index, 1, :, :]).reshape(1, PATCH_HEIGHT, PATCH_WIDTH)
            
        if self.transform is not None:
            inputs = self.transform(inputs)
            
        inputs = np.concatenate([mask, inputs], axis=0)
        return (inputs, output)

    def __len__(self):
        """
        * Description: fucntion for noticing the length of dataset
        
        * Output:
        - length (int)
        """
        return self.data.shape[0]
</code></pre>
<p>After I read separate <code>*.npy</code> files as a <code>PartialDataset</code> instance, I stored them in the list like below:</p>
<pre><code class="lang-auto">def construct_partial_dataset(read_path, test_date=['20190212', '20190612', '20190912', '20191216'], transform=transform):
    test_date = test_date
    training_list, test_list = [], []
    
    for path, dirs, files in os.walk(read_path):
        if dirs != []: continue 
        file_list = sorted(files)
        for file in file_list:
            if '.conf' in file: continue
                
            is_train = False if file[:8] in test_date else True 
            if is_train:
                training_list.append(PartialDataset(read_path=os.path.join(read_path, 'training'), date=file[:8], transform=transform))
            else:
                test_list.append(PartialDataset(read_path=os.path.join(read_path, 'validation'), date=file[:8], transform=transform))
    
    return training_list, test_list
</code></pre>
<p>After constructing two lists (<code>training_list</code> and <code>test_list</code>), I’ve followed your advice: <code>ConcatDataset</code> and <code>Subset</code> to construct a final training/test dataset containing whole corresponding arrays.</p>
<pre><code class="lang-auto">def concat_list_of_partial_datasets(dataset_list, num_samples=-1):
    """
    * Description: function of concatenating `PartialDataset` object to be a dataset using `torch.utils.data.ConcatDataset`
    
    * Argument:
    - dataset_list (list): list of `PartialDataset` instances
    
    * Output:
    - a dataset (torch.utils.data.ConcatDataset)
    """
    dataset = None
    for i in range(len(dataset_list) - 1):
        if i == 0:   
            idx1 = torch.randperm(len(dataset_list[i]))[:num_samples]
            idx2 = torch.randperm(len(dataset_list[i + 1]))[:num_samples]

            dataset1 = Subset(dataset_list[i], idx1)
            dataset2 = Subset(dataset_list[i + 1], idx2)
            dataset = ConcatDataset((dataset1, dataset2))
        else:
            idx = torch.randperm(len(dataset_list[i + 1]))[:num_samples]
            dataset_next = Subset(dataset_list[i + 1], idx)
            dataset = ConcatDataset((dataset, dataset_next))
    
    return dataset
</code></pre>
<p>Finally, I could utilize <code>DataLoader</code> for loading data!</p>
<pre><code class="lang-auto">training_set, test_set = concat_list_of_partial_datasets(dataset_list=training_list, num_samples=8192), concat_list_of_partial_datasets(dataset_list=test_list, num_samples=1024)

training_loader = DataLoader(training_set, batch_size=HYPERPARAMS['batch_size'], shuffle=True, num_workers=16)
test_loader = DataLoader(test_set, batch_size=HYPERPARAMS['batch_size'], shuffle=False, num_workers=16)
</code></pre>
<p>Since I am not that proficient in Python and PyTorch, still believe there exists more memory-efficient and better way to solve the situation like mine…!<br>
(Please let me know if my code has problems…)</p>
<p>Thank you again for your kind and gentle answer.<br>
Have a good day!</p>