Bigger batch_size increases training time

Norbert_Szorc: <p>During testing phase I noticed that in every model I have tested so far, after increasing batch_size training time increases too. Is it even possible? Maybe I am missing something in code?</p>

ptrblck: <p>Are you checking the iteration or epoch time?<br>
The interesting performance would be given as e.g. samples/second not the iteration time itself.</p>

Norbert_Szorc: <p>I was checking execution time of 5 epochs and it seemed weird that when I was trying to use all of my gpu memory, it was actually working twice slower than 1/4 max batch size. I do not mean it was learning slower, just the execution time of epoch. I dont know is that even an error, as far as I understand larger batches means squeezing more samples and averaging their losses to backtrack all of them at once, so theoretically larger batch should leads to shorten execution time. And batch_size=1 needs actually more time to do one epoch than batch_size=32, but although i have more memory in gpu the more I increase batch size from some point, the more its slowing down. I’m worried its because my hardware or some problem in code and Im not sure should it works like that.</p>
<p>Here are some of my results during ShuffleNet V2 x0.5 training:</p>
<p><em>Fine tuning / num_workers: 4 / Optimizer: SGD / Loss: Cross Entropy /  Data Augmentation: Rotate, Horizontal and Vertical Flip / Outputs: 3 / Input size: 224x224 / Epochs: 5 / Dataset size: 1500</em></p>
<p>Batch size: 1 <strong>|</strong> Training time: 184 s <strong>|</strong> Gpu usage: 1462 MB<br>
Batch size: 142 <strong>|</strong> Training time: 39 s <strong>|</strong> Gpu usage: 3591 MB<br>
Batch size: 284 <strong>|</strong> Training time: 47 s <strong>|</strong> Gpu usage: 5629 MB<br>
Batch size: 424 <strong>|</strong> Training time: 53 s <strong>|</strong> Gpu usage: 7523 MB<br>
Batch size: 566 <strong>|</strong> Training time: 56 s <strong>|</strong> Gpu usage: 7770 MB</p>
<p>As u can see increasing batch size also increases total training time and this pattern is duplicated with other models.</p>

ptrblck: <p>Thanks for the update. To further isolate the bottleneck, you could remove the data loading part and use pre-allocated tensors in the posted batch sizes and profile the code again.<br>
If you are seeing the expected speedup, the bottleneck might be coming from the data loading itself.</p>

Norbert_Szorc: <p>Thanks for Your answer!</p>
<p>As u suggested, I changed <strong>getitem</strong> function in Dataloader class to something like this:</p>
<pre><code class="lang-auto">self.tensor = torch.ones((3, 224, 224))

    def __getitem__(self, idx):
        """imagePath = self.new_dataset + self.imgs[idx]["path"]
        image = Image.open(imagePath)
        image = self.transform(image)
        imageClass = self.imgs[idx]["label"]
        label = self.classes.index(imageClass)
        return image, label"""
        return self.tensor,2
</code></pre>
<p>But unfortunately the results are the same as before. Increasing batch size still increases total training time. Here are some tested examples.</p>
<p><strong>ShuffleNet V2 x0.5:</strong><br>
Batch size: 142 <strong>|</strong> Training time: 16,15 s <strong>|</strong><br>
Batch size: 284 <strong>|</strong> Training time: 16,71 s <strong>|</strong><br>
Batch size: 424 <strong>|</strong> Training time: 16,85 s <strong>|</strong><br>
Batch size: 560 <strong>|</strong> Training time: 17,15 s <strong>|</strong></p>
<p><strong>MobileNet V3 Small:</strong><br>
Batch size: 96 <strong>|</strong> Training time: 16,78 s <strong>|</strong><br>
Batch size: 194 <strong>|</strong> Training time: 16,91 s <strong>|</strong><br>
Batch size: 290 <strong>|</strong> Training time: 17,11 s <strong>|</strong><br>
Batch size: 386 <strong>|</strong> Training time: 17,61 s <strong>|</strong></p>
<p>Maybe it’s not even an error, but I thought that bigger batch size always decreases execution time.</p>

ptrblck: <p>Yes, your assumption is correct as the performance measures in samples/sec should increase in the optimal case and the epoch time should thus be lower, as seen in e.g. <a href="https://github.com/NVIDIA/DeepLearningExamples/tree/d788e8d4968e72c722c5148a50a7d4692f6e7bd3/PyTorch/Classification/ConvNets/efficientnet#tf32-inference-latency">Efficientnet-b0</a>.<br>
To further isolate the bottlenecks, you could profile the code with e.g Nsight Systems.</p>

Norbert_Szorc: <p>I used Pytorch profiler to profile training and tensorboard to make it looking cleaner. I have tested densenet121 with batch_size 20 and 60. Here are some graphs that profiler has created.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/b/7b45a1673ca4d1154aa3b36d7f3a7fed1e8df9db.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/7b45a1673ca4d1154aa3b36d7f3a7fed1e8df9db" title="densenet121"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/b/7b45a1673ca4d1154aa3b36d7f3a7fed1e8df9db_2_690x388.jpeg" alt="densenet121" data-base62-sha1="hAvUvK7StMhfwMLrbeqFVtVeaa7" width="690" height="388" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/b/7b45a1673ca4d1154aa3b36d7f3a7fed1e8df9db_2_690x388.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/7/b/7b45a1673ca4d1154aa3b36d7f3a7fed1e8df9db_2_1035x582.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/7/b/7b45a1673ca4d1154aa3b36d7f3a7fed1e8df9db_2_1380x776.jpeg 2x" data-dominant-color="EAECF0"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">densenet121</span><span class="informations">1920×1080 163 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>And as you can see, for batch_size=20 GPU usage was around 50% and average step time was almost two times faster than average step time for batch_size=60, even though the GPU consumption was lower, which confuses me.</p>
<p>Logs shows that kernel (which as far as I understand is GPU)  takes the biggest part of time and I don’t know whats the problem. If its not I/O that take time, neither data augmentation, why its still remains slower for bigger batches?</p>

Norbert_Szorc: <p>Its noteworthy that logs from testing VGG models looks a little bit different. Kernel execution time was the same no matter what batch size was (around 70%), but still with bigger batch size takes more time per step to execute…</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/4/e45b70527e95a9fb1bf0d336dab888b82da5d7af.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/e45b70527e95a9fb1bf0d336dab888b82da5d7af" title="VGG16"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/4/e45b70527e95a9fb1bf0d336dab888b82da5d7af_2_690x388.jpeg" alt="VGG16" data-base62-sha1="wA8LJuC6S7FzKcu5S0nD09yAAvZ" width="690" height="388" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/4/e45b70527e95a9fb1bf0d336dab888b82da5d7af_2_690x388.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/e/4/e45b70527e95a9fb1bf0d336dab888b82da5d7af_2_1035x582.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/e/4/e45b70527e95a9fb1bf0d336dab888b82da5d7af_2_1380x776.jpeg 2x" data-dominant-color="EBEDF0"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">VGG16</span><span class="informations">1920×1080 143 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>As previously described, it’s expected that the kernel times for larger batches take more time (as more work has to be done), but the epoch time should decrease, since more samples/sec should be processed.<br>
Based on the posted profiles:</p>
<ul>
<li>
<p>Densenet121: (20 / 42.710) = 0.468</p>
</li>
<li>
<p>Densenet121: (60 / 70.324) = 0.853</p>
</li>
<li>
<p>VGG16: (20 / 50.103) = 0.399</p>
</li>
<li>
<p>VGG16: (60 / 120.996) = 0.495</p>
</li>
</ul>
<p>so in both cases increasing the batch size should yield a higher throughput measured in samples/sec (or <code>us</code> in this case) assuming the profile is correct.</p>

Norbert_Szorc: <p>I did more tests to isolate the bottleneck even more. This time I changed code a little bit to navigate the times manually. Here is train function code:</p>
<blockquote>
<pre><code>def train(self, dataloader, model, loss_func, optimizer):

    train_time = time.time()
    data_time = time.time()
    total_data_time = 0
    total_pred_time = 0
    total_loss_time = 0
    total_zero_grad_time = 0
    total_backward_time = 0
    total_step_time = 0
    total_sample_time = 0

    for batch, (imgs, labels) in enumerate(dataloader):
        total_data_time += time.time() - data_time
        sample_time = time.time()

        imgs, labels = imgs.to(self.device), labels.to(self.device)

        pred_time = time.time()
        predictions = model(imgs)
        total_pred_time += time.time() - pred_time

        loss_time = time.time()
        loss = loss_func(predictions, labels)
        total_loss_time += time.time() - loss_time

        zero_grad_time = time.time()
        optimizer.zero_grad()
        total_zero_grad_time += time.time() - zero_grad_time

        backward_time = time.time()
        loss.backward()
        total_backward_time += time.time() - backward_time

        step_time = time.time()
        optimizer.step()
        total_step_time += time.time() - step_time

        total_sample_time += time.time() - sample_time
        data_time = time.time()

    print(f"Dataset: {len(dataloader.dataset)}")
    print(f"Dataloader: {len(dataloader)}")
    print(f"Train time: {time.time() - train_time} s")
    print(f"Sample time: {total_sample_time} s")
    print(f"Data time: {total_data_time} s")
    print(f"Pred time: {total_pred_time} s")
    print(f"Loss time: {total_loss_time} s")
    print(f"Zero time: {total_zero_grad_time} s")
    print(f"Backward time: {total_backward_time} s")
    print(f"Step time: {total_step_time} s")
</code></pre>
</blockquote>
<p>I did two tests on ShuffleNet V2 x0.5 with batch size 142 and 566. I chose this model because the dependence on it is the most visible. Here are the results:</p>
<p><strong>For batch_size=566:</strong><br>
Dataset: 1188<br>
Dataloader: 3<br>
Train time: 4.74889 s<br>
Total Sample time: 0.20170 s<br>
Total Data time: <strong>4.49197 s</strong><br>
Total Prediction time: 0.06613 s<br>
Total Loss time: 0.00386 s<br>
Total ZeroGrad time: 0.00027 s<br>
Total Backward time: 0.00483 s<br>
Total Step time: 0.00048 s</p>
<p><strong>For batch_size=142:</strong><br>
Dataset: 1188<br>
Dataloader: 9<br>
Train time: 2.70721 s<br>
Total Sample time: 0.20925 s<br>
Total Data time: <strong>2.44183 s</strong><br>
Total Prediction time: 0.09259 s<br>
Total Loss time: 0.00435 s<br>
Total ZeroGrad time: 0.00082 s<br>
Total Backward time: 0.00579 s<br>
Total Step time: 0.00109 s</p>
<p>I named it data time, but I dont know for sure whats going on between one iteration and another. I suspect its time of loading single batch from dataloader, but then why profiler shows it’s only small part of time?</p>

ptrblck: <p>CUDA operations are executed asynchronously, so you would have to synchronize the code via <code>torch.cuda.synchronize()</code> before starting and stopping the timers. Alternatively, you might want to use <code>torch.utils.benchmark</code>, which would add warmup iterations and synchronize for you.</p>

tao_wang2: <p><a class="mention" href="/u/norbert_szorc">@Norbert_Szorc</a> * Have you solved your problem?   I had the same problem <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=12" title=":sweat_smile:" class="emoji" alt=":sweat_smile:" loading="lazy" width="20" height="20"></p>

Norbert_Szorc: <p>Nah, we weren’t able to fix that problem <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=12" title=":confused:" class="emoji" alt=":confused:" loading="lazy" width="20" height="20"> It was a project that company gave as for an internship program, so after one month we just shared the results with them and dropped it.</p>

Leo_Huang: <p>I encoutered the some problem. Solved by increasing num_worker in dataloader to 10, or even more.</p>

thk_rohan: <p><a class="mention" href="/u/tao_wang2">@tao_wang2</a> <a class="mention" href="/u/leo_huang">@Leo_Huang</a> Have you all been able to solve this issue. Whenever I increase my batch size my iteration speed is dropping and also the time for an epoch is increasing. I even tried increasing the number of workers. Still it’s not showing any significant change.</p>

Leo_Huang: <p>My case is my bottleneck is loading the data from disk. Therefore increasing num worker solved the problem for me. Sometimes I preload all data into mem, and if the size fit, it always help the speed</p>

thk_rohan: <p>I also tried this approach still didn’t gave any improvement. I am working using transformers architecture. I am also wondering whether its because of the model architecture by itself.</p>