DataLoader not workin properly

Rohan_Bareja: <p>I am using custom dataset class and then using data loader to load my data.</p>
<pre><code class="lang-auto">train_loader = torch.utils.data.DataLoader(
        dataset_train,
        sampler=train_sampler,
        batch_size=args.batch_size_per_gpu,
        num_workers=args.num_workers,
        pin_memory=True,
    )
    print(f"Data loaded with {len(dataset_train)} train imgs.")
</code></pre>
<p>The length of the dataset is 85 images.</p>
<p><code>print("for training model,size of train loader", len(train_loader.dataset))</code></p>
<p>But when I iterate through the data loader, it only divides into 2 batches of 16 and 6 images (batch size of 16) so only 22 images.I don’t know what happens with the rest of the images.I am using DINO tool’s eval_linear script <a href="https://github.com/facebookresearch/dino/blob/main/eval_linear.py" rel="noopener nofollow ugc">https://github.com/facebookresearch/dino/blob/main/eval_linear.py</a>.</p>
<p>I had to modify that code for my custom dataset as well.So it looks like this:</p>
<pre><code class="lang-auto">def train(model, linear_classifier, optimizer, train_loader, epoch, n, avgpool):
    linear_classifier.train()
    metric_logger = utils.MetricLogger(delimiter="  ")
    metric_logger.add_meter('lr', utils.SmoothedValue(window_size=1, fmt='{value:.6f}'))
    header = 'Epoch: [{}]'.format(epoch)
    all_labels = []


    for it,sample in enumerate(metric_logger.log_every(train_loader, 10)):
   
        #print('type of sample is ...',sample)
        images = sample['image']
 
        ## extract label
        index = sample['label']
        all_labels.append(index)
        print('label is ...',index)
        # move to gpu
        images = images.cuda(non_blocking=True)
        ## index is the target/label value
        index = index.cuda(non_blocking=True)
</code></pre>
<p>Interestingly the code works perfectly fine with torch version 1.7.0 and torch metrics 0.8.2 but due to GPU incompatibility , I needed to upgrade to both tools and this throws weird challenge.</p>
<p>I even provide batch size of 85 but it only iterates through 22 images.The length of the dataset in the data loader just before iterating is 85 images, so I don’t know why the rest of the images are not trained and the batches are not divided accordingly.There should be 5 batches with 16 images and the last batch with 5 images, so total 6 batches.</p>
<p>I also tried changing the print_freq of metric logger but it also did not help.</p>
<p>Can anyone provide any suggestion?</p>

Rohan_Bareja: <p>I tried checking the number of batches using the code in this post.</p>
<aside class="quote quote-modified" data-post="5" data-topic="64058">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/dataloader-length-1-instead-of-number-of-images/64058/5">Dataloader length==1 instead of number of images</a> <a class="badge-category__wrapper " href="/c/vision/5"><span data-category-id="5" data-drop-close="true" class="badge-category " title="Topics related to either pytorch/vision or vision research related topics"><span class="badge-category__name">vision</span></span></a>
  </div>
  <blockquote>
    Could you try to narrow down your code to just the data loading, as you shouldn’t get a single batch out of your DataLoader using the specified arguments. 
Have a look at this code snippet: 
N = 4000
batch_size = 16
dataset = TensorDataset(
    torch.randn(N, 1)
)
loader = DataLoader(
    dataset,
    batch_size=batch_size
)

print('len(dataset) ', len(dataset))
print('len(loader) ', len(loader))
print('expected number of batches {}'.format(
    math.ceil(N / batch_size)))

counter = 0
for data …
  </blockquote>
</aside>

<pre><code class="lang-auto">print('expected number of batches {}'.format(
    math.ceil(N / batch_size)))
</code></pre>
<p>And it printed out 6 batches as well.So not sure where the rest of the batches go?</p>
<p>Also I checked another example when I load 573 images, 144 images are trained(batch size 16, 9 batches). This equals 144 images. If we look at the ratio 144/573 its 25% approx, so is the ratio of 22/85 = 25%. So the rest of 75% images how can get those back?</p>

ptrblck: <p>The length of the dataset (and thus total number of samples) is irrelevant if you provide a custom sampler.<br>
Here is an example:</p>
<pre><code class="lang-python">dataset = torch.utils.data.TensorDataset(torch.randn(100, 1))
print(len(dataset))
# 100

loader = DataLoader(dataset, batch_size=10)
print(len(loader.dataset))
# 100
print(len(loader.sampler))
# 100
print(len(loader))
# 10

class MySampler(torch.utils.data.sampler.Sampler):
    def __init__(self, length):
        self.length = length
        
    def __iter__(self):
        return iter(range(self.length))
    
    def __len__(self):
        return self.length

sampler = MySampler(1)
loader = DataLoader(dataset, batch_size=10, sampler=sampler)
print(len(loader.dataset))
# 100
print(len(loader.sampler))
# 1
print(len(loader))
# 1
</code></pre>
<p>so check how <code>train_sampler</code> is defined.</p>

Rohan_Bareja: <p>So I think I am not using custom sampler, This is how is defined:<br>
<code>train_sampler = torch.utils.data.distributed.DistributedSampler(dataset_train)</code></p>
<p>How can I change this so as to it uses all of the images?</p>

ptrblck: <p>What is the length of the sampler?<br>
<code>DistributedSampler</code> splits the dataset using the world size so that each rank only loads its subset.</p>

Rohan_Bareja: <p>Also if I check the length of my loader itself its returned 2 when it should be 6, in the case of 85 images with batch size 16.So I don’t think sampler affects this, it might be the issue with loader?</p>

ptrblck: <p>No, as already explained and seen in my code snippet. I explicitly defined the sampler to have a length of <code>1</code> which then also returns a single batch containing a single sample. You can copy/paste my code snippet and run it or you could just check the <code>len</code> of the <code>train_sampler</code>, which will most likely be smaller than 85 images.</p>

Rohan_Bareja: <p>So the length of the train_sampler is 1.</p>