Dataloading of video sequences takes much longer with DistributedSampler

kzaitse: <p>Hi,<br>
I work with videos and have a dataloader that fetches short 3-frame sequences for training and full 150-frame sequences for validation. While the former is usually fast, although, in the DDP regime, it may take a while to initialize the Dataloader, the latter is much slower compared to the non-DDP version. Non-DDP dataloading takes ~1.5s per sequence, while the DDP one is ~4min. Of these 4min, most of the time is spent on dataloader initialization, where everything is frozen. When the iteration starts, each step in the loop is also slower than in the non-DDP setting, but not significantly. Adding more GPUs in a single-node setup makes the problem worse.<br>
Here is my setup:</p>
<pre><code class="lang-auto">        train_sampler = DistributedSampler(train_dataset, num_replicas=world_size, rank=rank)
        train_loader = DataLoader(
            train_dataset,
            batch_size=args.batch_size,
            sampler=train_sampler,
            collate_fn=collate_fn,
            num_workers=args.num_workers,
        )
        val_sampler = DistributedSampler(val_dataset, num_replicas=world_size, rank=rank)
        val_batch_size = min(max(1, args.num_workers), args.batch_size)
        val_loader = DataLoader(
            val_dataset,
            batch_size=val_batch_size,
            sampler=val_sampler,
            collate_fn=collate_fn,
            num_workers=args.num_workers,
        )
</code></pre>
<p>, where <code>args.num_workers=4</code>. In the non-DDP setup, the <code>val_loader</code> looks like this:</p>
<pre><code class="lang-auto">val_loader = DataLoader(
    val_dataset,
    batch_size=val_batch_size,
    shuffle=False,
    collate_fn=collate_fn,
    num_workers=args.num_workers,
)
</code></pre>
<p>Every epoch I iterate over each loader once.<br>
I tried <code>persistent_workers=True</code>, <code>pin_memory=True</code>, decreasing <code>val_batch_size</code>, setting <code>num_workers=0</code>, <code>NCCL_P2P_DISABLE=1</code>, but it didn’t improve the dataloading speed for the validation set. While I could change the way I load validation data, I would like to know if it is possible to optimize the current setup.<br>
My questions are on how I could further optimize my setup to reduce dataloading freezes during validation and how to write optimal dataloaders for the distributed mode in general?</p>

ptrblck: <aside class="quote no-group" data-username="kzaitse" data-post="1" data-topic="215813">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kzaitse/48/58706_2.png" class="avatar"> kzaitse:</div>
<blockquote>
<p>Of these 4min, most of the time is spent on dataloader initialization, where everything is frozen.</p>
</blockquote>
</aside>
<p>I would assume copying the <code>Dataset</code> might take the majority of the time. Are you eagerly pre-loading all samples or are you using lazy loading?</p>

kzaitse: <p>Samples are lazy-loaded within <code>__getitem__</code>. At initialization, it reads several files from the disk, each up to a few KB in size.<br>
I also experimented with multithreading within <code>__getitem__</code> for loading long sequences. It gives a substantial speedup in other cases, except for the one described.</p>

ptrblck: <aside class="quote no-group" data-username="kzaitse" data-post="3" data-topic="215813">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/kzaitse/48/58706_2.png" class="avatar"> kzaitse:</div>
<blockquote>
<p>At initialization, it reads several files from the disk, each up to a few KB in size.</p>
</blockquote>
</aside>
<p>Try to profile this section or for the sake of debugging, replace the actual reads from disk with generating fake or static data to see if maybe the disk reads slow down the worker initialization.</p>