Custom DataLoader unexpected behaviour

bornexmachina: <p>I am trying to combine codes of two repos.<br>
From the first I get the data &amp; its dataloader. From the second the models.</p>
<p>Now in my custom loader I have</p>
<pre><code class="lang-auto">train_index, valid_index = train_test_split(range(len(train_data)), test_size=args.validation_split)
        self.train_set = Subset(train_data, train_index)
        self.train_loader = torch.utils.data.DataLoader(
            self.train_set,
            batch_size=batch_size, shuffle=shuffle, batch_sampler=batch_sampler,
        )
        self.valid_dataset   = Subset(train_data, valid_index)
        self.valid_loader = torch.utils.data.DataLoader(
            self.valid_dataset,
            batch_size=batch_size, shuffle=False, batch_sampler=batch_sampler,
        )

        self.test_loader = []
        for id in self.section_id_list:
           _test_loader = DCASE202XT2Loader(
                data_path,
                dataset_name=dataset_name,
                section_keyword=section_keyword,
                machine_type=machine_type,
                train=False,
                section_ids=[id],
                frames=args.frames,
                n_mels=args.n_mels,
                frame_hop_length=args.frame_hop_length,
                n_fft=args.n_fft,
                hop_length=args.hop_length,
                power=args.power,
                fmax=args.fmax,
                fmin=args.fmin,
                win_length=args.win_length,
                data_type=data_type,
                is_auto_download=args.is_auto_download,
           )

           self.test_loader.append(
                torch.utils.data.DataLoader(
                    _test_loader,
                    batch_size=_test_loader.n_vectors_ea_file, shuffle=False
                )
           )
</code></pre>
<p>later I get the loaders as</p>
<pre><code class="lang-auto">
# Get train data loader
train_loader = dataset.train_loader
...
# Get test data loader
test_loader = dataset.test_loader
</code></pre>
<p>Now in training I go in a loop and get the input. Works well, my training starts.</p>
<pre><code class="lang-auto">for batch_idx, data in enumerate(tqdm(train_loader)):
    inputs = data[0]
</code></pre>
<p>In my test I would like to do the same,  but I get the “DataLoader not subscritable” error.<br>
Then I assumed, that as I get a DataLoader and my custom <strong>get_item</strong> has five return values, I need to unpack.</p>
<pre><code class="lang-auto">for batch_idx, data in enumerate(tqdm(test_loader)):
    input, label, _, _, idx = data
</code></pre>
<p>But this also results in “two many values … expected 2” Error. Thus data should not be unpacked.</p>
<p>I am a bit lost how to get my test loader running.</p>
<p>Should I subscript it when I assign</p>
<pre><code class="lang-auto">
test_loader = self.test_loader[0] as self.test_loader is actually a list?
</code></pre>
<p>Also could I go for this approach in my trainloader? Would look a bit more sexy</p>
<pre><code class="lang-auto">for batch_idx, data in enumerate(tqdm(train_loader)):
    inputs, _, _, _, _ = data
</code></pre>

ptrblck: <p>You should iterate a <code>DataLoader</code> instance instead of a <code>list</code> containing <code>DataLoader</code>s unless you want to use a nested loop and get the data of each <code>test_loader</code>.</p>
<pre><code class="lang-python">dataset = TensorDataset(torch.randn(100, 1), torch.randn(100, 1))
loader = DataLoader(dataset, batch_size=50)

for idx, (data, target) in enumerate(loader):
    print(data.shape)
# torch.Size([50, 1])
# torch.Size([50, 1])

test_loader = []
for _ in range(3):
    test_loader.append(DataLoader(dataset, batch_size=50))

for i, loader in enumerate(tqdm(test_loader)):
    for idx, (data, target) in enumerate(loader):
        print(data.shape)
# torch.Size([50, 1])
# torch.Size([50, 1])
# torch.Size([50, 1])
# torch.Size([50, 1])
# torch.Size([50, 1])   
</code></pre>