Data Loading takes very long time with Docker compared to local Pytorch without Docker

Hussein_A_Hassan: <p>Hello</p>
<p>I have a training script when I run it on my local machine, the loading time of the data for one epoch is around 30 minutes but when I run the same script on much powerful server with Docker the loading time takes around 5 hours.</p>
<p>I don’t have much experience with Docker. I used the following Dockerfile to build the image.</p>
<p>FROM <a href="http://nvcr.io/nvidia/pytorch:21.08-py3" rel="noopener nofollow ugc">nvcr.io/nvidia/pytorch:21.08-py3</a></p>
<p>ENV DEBIAN_FRONTEND=noninteractive<br>
RUN apt-get update &amp;&amp; apt-get install -y ffmpeg<br>
RUN pip install pandas<br>
RUN pip install scikit-video<br>
RUN pip install ffmpeg-python<br>
RUN pip install scikit-learn<br>
RUN pip install opencv-python<br>
RUN pip install tqdm<br>
RUN pip install torchsummary<br>
RUN pip install tensorboardX<br>
CMD [“python”, “./train_classifier.py”]</p>
<p>and I run the Docker container with the following command:</p>
<p>sudo docker run --ipc=host -it --rm --gpus=device=0 --name train_container --network=host -v /home/h/data/:/workspace train_image:3.0 bash</p>
<p>I checked the shared memory of the server and it has over 50% free space during training.<br>
In addition, when I set the number of workers in the training script to 0 no noticeable difference in the loading time would be achieved.</p>
<p>Please, if you have any suggestion to solve the problem let me know.</p>
<p>Your help is much appreciated.</p>

Michal_Bogacz: <p>Hello,<br>
Are those 5h together with container build, or the script execution have just slowed down so much. Good practice is to define the versions of every package, build process would be much faster.<br>
If the slow-down is only the code slowdown, can you show what happens in code, how do you load data?</p>

Hussein_A_Hassan: <p>Thanks <a class="mention" href="/u/michal_bogacz">@Michal_Bogacz</a>  for your help.</p>
<p>The script execution is just went extremely slow, The 5h is the time to load the data and do some augmentations on it only NOT the container image build.</p>
<p>I am trying to re run the training of the following paper:</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://arxiv.org/abs/2101.07974">
  <header class="source">
      <img src="https://static.arxiv.org/static/browse/0.3.4/images/icons/favicon.ico" class="site-icon" width="32" height="32">

      <a href="https://arxiv.org/abs/2101.07974" target="_blank" rel="noopener nofollow ugc">arXiv.org</a>
  </header>

  <article class="onebox-body">
    <img src="https://static.arxiv.org/icons/twitter/arxiv-logo-twitter-square.png" class="thumbnail onebox-avatar" width="500" height="500">

<h3><a href="https://arxiv.org/abs/2101.07974" target="_blank" rel="noopener nofollow ugc">TCLR: Temporal Contrastive Learning for Video Representation</a></h3>

  <p>Contrastive learning has nearly closed the gap between supervised and
self-supervised learning of image representations, and has also been explored
for videos. However, prior work on contrastive learning for video data has not
explored the effect of...</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>This is the data loader code from the paper:</p><aside class="onebox githubblob" data-onebox-src="https://github.com/DAVEISHAN/TCLR/blob/main/tclr_pretraining/dl_tclr.py">
  <header class="source">

      <a href="https://github.com/DAVEISHAN/TCLR/blob/main/tclr_pretraining/dl_tclr.py" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/DAVEISHAN/TCLR/blob/main/tclr_pretraining/dl_tclr.py" target="_blank" rel="noopener nofollow ugc">DAVEISHAN/TCLR/blob/main/tclr_pretraining/dl_tclr.py</a></h4>


      <pre><code class="lang-py">r'''This dataloader is an attemp to make a master DL that provides 2 augmented version
of a sparse clip (covering minimum 64 frames) and 2 augmented versions of 4 dense clips
(covering 16 frames temporal span minimum)'''
import os
import torch
import numpy as np
import matplotlib.pyplot as plt
from torch.utils.data import Dataset, DataLoader
from torchvision import transforms, utils
import config as cfg
import random
import pickle
import parameters as params
import json
import math
import cv2
# from tqdm import tqdm
import time
import torchvision.transforms as trans
# from decord import VideoReader
</code></pre>



  This file has been truncated. <a href="https://github.com/DAVEISHAN/TCLR/blob/main/tclr_pretraining/dl_tclr.py" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>I suspect that it could be a limitation on reading speed from the SSD or maybe the<br>
cpu speed. I don’t know exactly but it could that there are some restrictions made by<br>
the Docker.</p>
<p>Many Thanks <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

ptrblck: <aside class="quote no-group" data-username="Hussein_A_Hassan" data-post="3" data-topic="151562">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hussein_a_hassan/48/44774_2.png" class="avatar"> Hussein_A_Hassan:</div>
<blockquote>
<p>I don’t know exactly but it could that there are some restrictions made by<br>
the Docker.</p>
</blockquote>
</aside>
<p>Did you test the data loading on the server without using docker as it seems you think this issue is docker-related?</p>