How to use part of the samples from data_loader

username: Haolin_Jia
cooked: <p>this is my data_loader part</p>
<pre><code class="lang-python">dataset = torchvision.datasets.ImageFolder('/xxxxxx/coco/', transform=data_transform)
data_loader = torch.utils.data.DataLoader(dataset, batch_size=batch_size, shuffle=True)
</code></pre>
<p>but as you know, the COCO dataset is large, and in my debugging stage, I prefer to use part of the image from COCO, so I try this:</p>
<pre><code class="lang-python">dataset = torchvision.datasets.ImageFolder('/xxxxx/coco/', transform=data_transform)
dataset = torch.utils.data.Subset(dataset, [i for i in range(100)])
data_loader = torch.utils.data.DataLoader(dataset, batch_size=batch_size, shuffle=True)
</code></pre>
<p>I try to use <code>Subset</code> to extract the first 100 images, but fail:</p>
<pre><code class="lang-auto">File "/xxxxxxxxx/train2.py", line 120, in &lt;module&gt;
    content_img, _ = data_loader[num]
TypeError: 'DataLoader' object does not support indexing
</code></pre>
<p>This is part of my training code:</p>
<pre><code class="lang-python">for epoch in range(iter_times):
    print("Epoch %d" % epoch)

    with tqdm(enumerate(data_loader), total=100, ncols=40) as pbar:
        for batch, (content_imgs, _) in pbar:

            optimizer.zero_grad()
            # the rest
</code></pre>
<p>I know it easy to insert <code>if batch &gt;= 100: break</code> in the loop to solve this problem, but I think it is a bit awkward, so could you help with my trivial problem?<br>
What’s more, I guess <code>data_loader</code> is an iterator, so how can I get an sample of specified index from the data_loader? such as <code>data_loader[3]</code> it is convenient to test my code.<br>
I am a freshman using pytorch, thanks for your patience!</p>

username: InnovArul
cooked: <p>Perhaps you could try with <code>SubsetRandomSampler</code>.</p>
<p><a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.SubsetRandomSampler" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/docs/stable/data.html#torch.utils.data.SubsetRandomSampler</a></p>

username: Cheng_Zhang
cooked: <p>This is a little bit late, but I hope this can help the others.</p>
<p>In case if you can’t use <code>SubsetRandomSampler</code>, here’s a manual solution for you. (The reason why we can’t use <code>SubsetRandomSampler</code> is because we also need other samplers and two of them won’t work together) You can refer to this <a href="https://discuss.pytorch.org/t/dataloader-using-subsetrandomsampler-and-weightedrandomsampler-at-the-same-time/29907/2">question</a> where people get confused too.</p>
<p>If you look closely, the <code>image paths</code> and the <code>associated labels</code> are saved in <code>dataset.samples</code> from ImageFolder output. Therefore, you can do</p>
<pre><code class="lang-auto">dataset.samples = [dataset.samples[idx] for idx in your_list_of_interest]
dataset.targets = [dataset.targets[idx] for idx in your_list_of_interest]
dataset.targets = [dataset.targets[idx] for idx in your_list_of_interest]
</code></pre>
<p>to subsample</p>
<p>In your case, <code>your_list_of_interest</code> is <code>range(100)</code>.</p>