How to accelerate each epoch?

username: sriharsha0806
cooked: <p>Hi,</p>
<p>I am trying to build a face verification model using a siamese model on wikifaces dataset. Link to repo <a href="https://github.com/sriharsha0806/Face-Verification" rel="nofollow noopener">https://github.com/sriharsha0806/Face-Verification</a></p>
<p>I built a custom dataset and currently, when running train.py each epoch is taking too long. The dataset contains 2,24,800 images. I used only wiki faces which are 60,000 images in total. Still, it is taking more than 1-2 hours to complete each epoch. May I know what is the bug in the code or how to accelerate it?</p>

username: raghavendragaleppa
cooked: <p>Can you please post the output while training. Post the timing for each epoch also</p>

username: sriharsha0806
cooked: <p>for reference, I ran the experiment on my device with the below mentioned configurations<br>
This is the output and time taken by 100 iterations in an each epoch. In each iteration, there are 4 examples.<br>
Configuration of device: GTX970<br>
batch_size = 4<br>
number_of_workers = 4</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/6/86c2b72a71e10f2979313e8de04ca34d4d5a89e5.png" data-download-href="https://discuss.pytorch.org/uploads/default/86c2b72a71e10f2979313e8de04ca34d4d5a89e5" title="Screenshot%20from%202019-10-28%2018-22-38"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/6/86c2b72a71e10f2979313e8de04ca34d4d5a89e5_2_690x388.png" alt="Screenshot%20from%202019-10-28%2018-22-38" data-base62-sha1="je99JIXRQnw5weFbOw1f3dXViFD" width="690" height="388" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/6/86c2b72a71e10f2979313e8de04ca34d4d5a89e5_2_690x388.png, https://discuss.pytorch.org/uploads/default/optimized/3X/8/6/86c2b72a71e10f2979313e8de04ca34d4d5a89e5_2_1035x582.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/8/6/86c2b72a71e10f2979313e8de04ca34d4d5a89e5_2_1380x776.png 2x" data-dominant-color="EEEEEE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot%20from%202019-10-28%2018-22-38</span><span class="informations">1920×1080 217 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: raghavendragaleppa
cooked: <p>Hello. Maybe the FC layers are clogging up a lot of time. Remove the fully connected layers and replace them with a AveragePooling layer</p>

username: mailcorahul
cooked: <p>Hi, as <a class="mention" href="/u/raghavendragaleppa">@raghavendragaleppa</a> mentioned try using a fully convolutional network or a larger batch size(4 seems too low). But I doubt batch_size could be a bottleneck given the FC layers size.</p>

username: sriharsha0806
cooked: <p>Hi,  I am running main program with a batch size of 32</p>

username: sriharsha0806
cooked: <p>will try that but I think the main problem is with dataloader. I ran a dataiter</p>
<pre><code class="lang-auto">vis_dataloader = DataLoader(siamese_dataset,
                        shuffle=True,
                        num_workers=8,
                        batch_size=4)
from time import perf_counter
t1=perf_counter()
dataiter = iter(vis_dataloader)
example_batch = next(dataiter)
t2 = perf_counter()

concatenated = torch.cat((example_batch[0],example_batch[1]))
imshow(torchvision.utils.make_grid(concatenated))
print(example_batch[2].numpy())
</code></pre>
<p>It’s taking an average of 4s. In each epoch there are around 6000 iterations. I think this is where most of the time for code is elapsing. Anyway I will try out your idea and will let you.</p>

username: raghavendragaleppa
cooked: <p><a class="mention" href="/u/sriharsha0806">@sriharsha0806</a><br>
The num_workers=8 is a very high number if your CPU is not powerful enough and instead it will be an overhead, and besides num_worker doesn’t help much when your batch_size is low and you have less amount of augmentations going on. The dataloader works on CPU so keeping num_workers = 8 will be problematic if your CPU is not powerfull enough. Check the timing for each iteration for num_workers=1 and num_workers=0 and also check the timings by using pin_memory=True.</p>
<p>Also i can see that you have very less augmentation going on, so the problem must be in the dataloader itself. Reduce the num_workers to 1 and see how much each iteration takes.</p>

username: sriharsha0806
cooked: <p>Hi <a class="mention" href="/u/raghavendragaleppa">@raghavendragaleppa</a>  and <a class="mention" href="/u/mailcorahul">@mailcorahul</a></p>
<p>Actually, I forgot to mention net.train() before training This is what causing the lag. I apologise for the mistake.</p>