Using different custom dataloaders, with same index

Minseok: <p>Hi there,</p>
<p>I am currently drawing batch data from three same-shape(i.e., m by n matrix) independent data loaded on GPU all together, but due to the GPU memory usage problem I would like to write down my custom Dataset.</p>
<p>The thing is, I can pick item indices and use them in the three data sources before, but the dataloader actually does not provide any information about it. So my possible solution towards it is to modify <strong>getitem</strong> in my custom dataloader and use as below:</p>
<pre><code class="lang-auto">class MyDataset(Dataset):
    def __init__(self):
        self.cifar10 = datasets.CIFAR10(root='YOUR_PATH',
                                        download=False,
                                        train=True,
                                        transform=transforms.ToTensor())
        
    def __getitem__(self, index):
        data, target = self.cifar10[index]
        
        return data, target, index
</code></pre>
<p>which is from here: <code>[https://discuss.pytorch.org/t/how-does-one-obtain-indicies-from-a-dataloader/16847/3]</code></p>
<p>Here comes the question: I want to pick the indices from dataloader and use the indices to get items from other two dataloaders(the data shape are the same). Would there be more efficient way?</p>

ptrblck: <p>Could you explain a bit, what you mean by</p>
<aside class="quote no-group" data-username="Minseok" data-post="1" data-topic="53138">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/minseok/48/9774_2.png" class="avatar"> Minseok:</div>
<blockquote>
<p>I want to pick the indices from dataloader</p>
</blockquote>
</aside>
<p>?<br>
Would you like to determine the indices manually?</p>
<p>Would it work to initialize all three datasets inside your custom <code>MyDataset</code> and just index all three in <code>__getitem__</code> or am I misunderstanding the use case?</p>

Minseok: <p>I think I found a walkaround that initialize dataset independently and concatenate them in one dataset, then index it using the key information during the dataloading. Data from the same row comes out from datasets. The <code>shuffle=True</code> in the dataloader seems to shuffle all the concatenated datasets in the same shuffled indices. Specific code is as below and it works for me:</p>
<pre><code class="lang-auto">class ContextTensorDataset(Dataset):

    def __init__(self, context_type, directory_path, context_threshold=0, context_neglect=False):

        print(context_type + " Context Tensor Dataset initialized.")
        # Read data if the context is provided
        self.context_matrix = T.FloatTensor(np.load(directory_path + 'user_' + context_type + '_context.npy'))
        self.context_threshold = context_threshold
        self.context_type = context_type

        if context_neglect:
            self.context_matrix[self.context_matrix &lt; self.context_threshold] = 0

    def __getitem__(self, index):
        return {self.context_type: self.context_matrix[index]}

    def __len__(self):
        '''
        Returns the length of the data(i.e., number of users)
        '''
        return self.context_matrix.shape[0]

    def shuffle_index(self, new_index):
        self.context_matrix = self.context_matrix[new_index]

    @property
    def shape(self):
        return self.context_matrix.shape

class ConcatDatasets(Dataset):
    def __init__(self, *datasets):
        self.datasets = datasets

    def __getitem__(self, index):
        batch = {}
        for dataset in self.datasets:
            batch = {**batch, **dataset[index]}
        return batch

    def __len__(self):
        return min(len(d) for d in self.datasets)
</code></pre>
<p>thus the code in the main.py would be somewhat like below</p>
<pre><code class="lang-auto">    if 'G' in args.input_val:
        place_data = ContextTensorDataset('geo', directory_path, args.context, context_neglect=True)
        datasets += (place_data,)
    if 'T' in args.input_val:
        time_data = ContextTensorDataset('time', directory_path, args.context, context_neglect=True)
        datasets += (time_data,)
    if 'S' in args.input_val:
        seq_data = ContextTensorDataset('seq', directory_path, args.context, context_neglect=True)
        datasets += (seq_data,)


    dataloaders = DataLoader(ConcatDatasets(*datasets),
                            batch_size=args.batch_size,
                            shuffle=True,
                            pin_memory=True,
                            num_workers=4)

</code></pre>