About the relation between batch_size and length of data_loader

chenchr: <p>Hello. I want to ask about about the relation between batch_size and length of data_loader…<br>
Assuming I have a dataset with 1000 images and set as the train loader.<br>
During training there will be a python code like:</p>
<pre><code class="lang-auto">for i, (input, target) in enumerate(train_loader):
</code></pre>
<p>I not sure about that if I set the batchsize as 10, will the train_loader’s length be changed to 100? And it’s length become to 50 if I set the batchsize as 20?  Or no matter how I set the batchsize, the trainloader will keep it’s length 1000?<br>
Thank!</p>

yashcao: <p>I  am  not sure about your issues.<br>
then, I hold the view  that: if  the data_loader is 100 (constant),  and I set the batch_size = 20; I will know one iteration can fetch 20 data from data_loader, training all data  needs 100/20=5 iterations.</p>

chenchr: <p>But the number of backprop is different…</p>

ptrblck: <p>The length of the loader will adapt to the <code>batch_size</code>. So if your train dataset has 1000 samples and you use a <code>batch_size</code> of 10, the loader will have the length 100.<br>
Note that the last batch given from your loader can be smaller than the actual <code>batch_size</code>, if the dataset size is not evenly dividable by the <code>batch_size</code>. E.g. for 1001 samples, <code>batch_size</code> of 10, <code>train_loader</code> will have <code>len(train_loader)=101</code> and the last batch will only contain 1 sample. You can avoid this by setting <code>drop_last=True</code>.</p>
<pre><code>class MyDataset(Dataset):
    def __init__(self, size):
        self.x = torch.randn(size, 1)
    
    def __getitem__(self, index):
        return self.x[index]

    def __len__(self):
        return len(self.x)

dataset = MyDataset(1001)

data_loader = DataLoader(dataset,
                         batch_size=10)

len(data_loader)

for batch_idx, data in enumerate(data_loader):
    print 'batch idx{}, batch len {}'.format(
        batch_idx, len(data))

data_loader = DataLoader(dataset,
                     batch_size=10,
                     drop_last=True)

len(data_loader)

for batch_idx, data in enumerate(data_loader):
    print 'batch idx{}, batch len {}'.format(
        batch_idx, len(data))</code></pre>

chenchr: <p>Thanks for your reply. This is a bit different from caffe, which don’t have the concept of dataloader…</p>

jdhao: <p><code>Dataset</code> provide an interface to access single sample in dataset using its index. <code>Dataloader</code> is used to provide batch sample for training your models using SGD or similar variants.</p>

ptrblck: <p>And it’s beautiful isn’t it? <img src="//discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=5" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
Also <code>DataLoader</code> provides several arguments like <code>num_workers</code> to use multi processing for the data loading and preprocessing, <code>shuffle</code> etc.<br>
<a href="http://pytorch.org/tutorials/beginner/data_loading_tutorial.html" rel="nofollow noopener">This</a> might be a good starter for further questions.</p>

chenchr: <p>Thanks. <img src="//discuss.pytorch.org/images/emoji/apple/yum.png?v=5" title=":yum:" class="emoji" alt=":yum:"><img src="//discuss.pytorch.org/images/emoji/apple/yum.png?v=5" title=":yum:" class="emoji" alt=":yum:"> I think it’s more concrete than caffe.</p>

chenchr: <p>However, there will be some difference when reproduce something from caffe using pytorch. As caffe has not the concept of epoch, it update lr in every iter, pytorch update lr in every epoch, there will be some deviation…</p>

ptrblck: <p>What do you mean? The learning rate is usually not updated automatically.<br>
You can adjust the learning rate with e.g.:</p>
<pre><code>def adjust_learning_rate(optimizer, epoch):
    """Sets the learning rate to the initial LR decayed by 10 every 30 epochs"""
    lr = args.lr * (0.1 ** (epoch // 30))
    for param_group in optimizer.param_groups:
        param_group['lr'] = lr
</code></pre>
<p>or with <code>optim.lr_scheduler</code>.</p>

chenchr: <p>Yes, you are right. What I am say is that In the pytorch official example, usually there will be some code like this:</p>
<pre><code class="lang-auto">for epoch in range(args.start_epoch, args.epochs):
        if args.distributed:
            train_sampler.set_epoch(epoch)
        adjust_learning_rate(optimizer, epoch)

        # train for one epoch
        train(train_loader, model, criterion, optimizer, epoch)

        # evaluate on validation set
        prec1 = validate(val_loader, model, criterion)
</code></pre>
<p>from <a href="http://imagenet.py" rel="nofollow noopener">imagenet.py</a>.<br>
From the code above, usually pytorch update lr out of the loop over train_loader, but caffe update every iter. I can also change the code <code>adjust_learning_rate(optimizer, epoch)</code> to  <code>adjust_learning_rate(optimizer, iter)</code> and move it into the loop over train_loader too. But in many example code, I found that write in the former style.</p>

jdhao: <p>You can set up a counter to count how many iterations have passed, which is rather simple. Then you can adjust the learning rate base on the iteration, the same way as caffe.</p>
<p>I think adjust the learning rate based on epoch maybe enough to achieve good results.</p>

hesham_gabr: <p>Thnaks<br>
What is meaning of len(data_loader)? what length indicate?</p>

nivesh_gadipudi: <p>It returns the number of batches of data generated from DataLoader.<br>
For instance: if the total samples in your dataset is 320 and you’ve selected batch_size as 32, len(data_loader) will be 10, if batch_size is 16 len(data_loader) is 20.</p>
<p>to keep it simple, len(data_loader) = ceil((no. of samples in dataset)/batchsize)</p>

hesham_gabr: <p>Thnks<br>
i have a different batch size of training ,testing and validation dataset<br>
how make all the same batch size<br>
train_loader = torch.utils.data.DataLoader(train_data, batch_size=batch_size, num_workers=num_workers, shuffle=True)</p>
<p>valid_loader = torch.utils.data.DataLoader(valid_data, batch_size=batch_size, num_workers=num_workers, shuffle=True)</p>
<p>test_loader = torch.utils.data.DataLoader(test_data, batch_size=batch_size, num_workers=num_workers, shuffle=True)</p>
<p>i did that but the batch size is still differ</p>

nivesh_gadipudi: <p>Generally, you can declare batch_size variable before calling DataLoader for Test, train and val.This should work!</p>

Kapil_Rana: <p>In Simple words, train_loader will provide batches of images (in size of batch-size). So number of iteration  per epoch would be len(train_loader.dataset)/batch_size</p>

Kapil_Rana: <p>It is len(data_loader.dataset)/batchsize.</p>
<p>and len(data_loader.dataset) is  total number of images in dataset (which is passed in data_loader).</p>

Kapil_Rana: <p>I think it should work. It might be variable name issue. Try to give constant number.<br>
train_loader = torch.utils.data.DataLoader(train_data, batch_size=64, num_workers=num_workers, shuffle=True)</p>

Brock_Brown: <p>It seems like every time I want to know something niche about PyTorch I always see your face. Just wanna say thanks for the help. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>