Why my DistributedDataParallel is slower than DataParallel if my Dataset is not loaded fully in memory

ammary-mo: <p>I am using a custom dataset and used custom data.Dataset class for loading it.</p>
<pre><code class="lang-auto">class MyDataset(data.Dataset):

    def __init__(self, datasets, transform=None, target_transform=None):
        self.datasets = datasets
        self.transform = transform

    def __len__(self):
        return len(self.datasets)

    def __getitem__(self, index):
        image = Image.open(os.path.join(self.datasets[index][0]))
        if self.transform:
            image = self.transform(image)
        return image, torch.tensor(self.datasets[index][1], dtype=torch.long)
</code></pre>
<p>When I started training on my 4 GPU machine, unlike the mentioned in <a href="https://pytorch.org/tutorials/beginner/dist_overview.html" rel="noopener nofollow ugc">Pytorch documentation</a>, I found that DistributedDataParallel is slower than DataParallel!</p>
<p>I reviewed my code carefully and tried different configurations and batch sizes (especially the DataLoader num_workers) to see what makes DistributedDataParallel runs faster than DataParallel as expected, but nothing worked.</p>
<p>The only change I did that made DistributedDataParallel faster is loading the whole dataset into memory during initialization!</p>
<pre><code class="lang-auto">class Inmemory_Dataset(data.Dataset):

    def __init__(self, datasets, transform=None, target_transform=None):
        self.datasets = datasets
        transform = transform
        image_list = []
        target_list = []
        for i, data in enumerate(datasets):
            image = Image.open(os.path.join(data[0]))
            if transform:
                image = transform(image)
            image_list.append(image.numpy())
            target_list.append(data[1])
        self.images = torch.tensor(image_list)
        self.targets = torch.tensor(target_list, dtype=torch.long)

    def __len__(self):
        return len(self.datasets)

    def __getitem__(self, index):
        return self.images[index], self.targets[index]
</code></pre>
<p>After this change, DistributedDataParallel became 30% faster. but I do not think this is how it should be. Because what if my dataset does not fit into memory?</p>
<p>Below I highlight the main parts where I setup the use for both DataParallel and DistributedDataParallel. Notice that the overall effictive batch size is the same in both cases.</p>
<p>DataParallel:</p>
<pre><code class="lang-auto">    batch_size = 100
    if torch.cuda.device_count() &gt; 1:
        print("Using DataParallel...")
        model = nn.DataParallel(model)
        batch_size = batch_size * torch.cuda.device_count()
</code></pre>
<p>DistributedDataParallel:</p>
<pre><code class="lang-auto">def train(gpu, args):
    # print(args)
    rank = args.nr * args.gpus + gpu
    dist.init_process_group(backend='nccl', init_method='env://', world_size=args.world_size, rank=rank)
</code></pre>
<pre><code class="lang-auto">batch_size = 100
model = nn.parallel.DistributedDataParallel(model, device_ids=[gpu])
train_sampler = torch.utils.data.distributed.DistributedSampler(training_dataset,
                                                                    num_replicas=args.world_size,
                                                                    rank=rank)
training_dataloader = torch.utils.data.DataLoader(
        dataset=training_dataset,
        batch_size=batch_size,
        shuffle=False,
        num_workers=1,
        sampler=train_sampler,
        pin_memory=True)
</code></pre>

mrshenli: <p>Hey <a class="mention" href="/u/ammary-mo">@ammary-mo</a>, how did you measure the delay? Since <code>DataParallel</code> and <code>DistributedDataParallel</code> are only involved in the forward and backward passes,  could you please try using <a href="https://pytorch.org/docs/stable/cuda.html#torch.cuda.Event.elapsed_time" rel="noopener nofollow ugc"><code>elapsed_time</code></a> to measure data loading, forward and backward delay breakdowns? See the following discussion. It’s possible that if multiple DDP processes try to read from the same file, contentions might lead to data loading perf regression. If that’s the case, the solution would be implementing a more performant data loder.</p>
<aside class="quote" data-post="11" data-topic="87220">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mrshenli/48/12220_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/issue-with-using-dataparallel-includes-minimal-code/87220/11">Issue with using DataParallel (includes minimal code)</a> <a class="badge-category__wrapper " href="/c/distributed/12"><span data-category-id="12" data-drop-close="true" class="badge-category "><span class="badge-category__name">distributed</span></span></a>
  </div>
  <blockquote>
    Can we profile how much of the 434s are spent in the forward pass when DP is not present? And how much of that is spent on GPU? This can be measured using <a href="https://pytorch.org/docs/stable/cuda.html#torch.cuda.Event.elapsed_time" rel="noopener nofollow ugc">elapsed_time</a> . See <a href="https://discuss.pytorch.org/t/does-it-ever-make-sense-to-try-model-parallelism-even-if-the-model-fits/86432/14">this discussion</a>. 
Note that multi-thread cannot parallelize normal Python ops due to Python GIL, and the parallelism only kicks in when the execution does not require GIL (e.g., CPU/GPU ops that explicitly drops GIL).
  </blockquote>
</aside>

<p>cc <a class="mention" href="/u/vitalyfedyunin">@VitalyFedyunin</a> <a class="mention" href="/u/glaringlee">@glaringlee</a> for DataLoader and DataSampler.</p>

ammary-mo: <p>Hey <a class="mention" href="/u/mrshenli">@mrshenli</a></p>
<p>Thanks for tuning in, I understand why you are asking to measure the forward pass for both DP and DDP. But if you think about it from the end-user perspective, I care more about the overall training performance (which involves both model sync and data batches loading). If DDP provides better time only in the forward pass but (somehow) wastes the time it saved in data loading, then, as overall, DP will be a better option! Do not you agree?!</p>

Alexey_Demyanchuk: <p>Is it possible, that if your data is small enough to entirely fit into the memory, the DDP setup overhead is just increasing time on the task without any performance improvement? In other words: GPU utilization is small enough, you just can’t see the gains of using multiple GPUs</p>

ammary-mo: <aside class="quote" data-post="4" data-topic="115677">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alexey_demyanchuk/48/25254_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/why-my-distributeddataparallel-is-slower-than-dataparallel-if-my-dataset-is-not-loaded-fully-in-memory/115677/4">Why my DistributedDataParallel is slower than DataParallel if my Dataset is not loaded fully in memory</a> <a class="badge-category__wrapper " href="/c/distributed/12"><span data-category-id="12" data-drop-close="true" class="badge-category "><span class="badge-category__name">distributed</span></span></a>
  </div>
  <blockquote>
    Is it possible, that if your data is small enough to entirely fit into the memory, the DDP setup overhead is just increasing time on the task without any performance improvement? In other words: GPU utilization is small enough, you just can’t see the gains of using multiple GPUs
  </blockquote>
</aside>

<p>Good question <a class="mention" href="/u/alexey_demyanchuk">@Alexey_Demyanchuk</a>.</p>
<p>But my answer is no. My data is large to fit in GPU, that is why I started by loading it from disk. However, when I found that DDP is slower than DP, as I mentioned in the question, I started comparing both with different configurations to see what works. Eventually, the only change that made DDP faster is when I reduced my data size and loaded it into memory. I hope this clarifies the situation.</p>

Alexey_Demyanchuk: <p>My suggestion would be to profile the pipeline. Is GPU utilization near 100% throughout training? If it is not, you could have some sort of preprocessing bottleneck (I/O or CPU bound). Does it make sense in your case?</p>

ammary-mo: <p>Not sure why to check the GPU utilization? GPU utilization depends on multiple factors, like batch size and even image size.</p>
<p>Anyhow, I checked GPU utilization and it is low in all cases. but this is not due to data-loading bottleneck rather than because I am using small model and small data (MNIST-like data set) just to compare the performance.</p>

ammary-mo: <p><a class="mention" href="/u/mrshenli">@mrshenli</a>, it seems you are right. It looks like a dataloading issue to me.</p>
<p>In DDP, with only two workers (num_workers=2), it is clear that data loading time is a bottleneck as one worker/batch is constantly taking more time to load than the other:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/4/b/4b7ec6b8b0f5cbe50b95e86c7534768424f08834.png" alt="image" data-base62-sha1="aLRqtZTk1F33F8SG8UdgVNmjWqo" width="515" height="249"></p>
<p>To solution is to increase the num_workers to the limit that hides the dataloading delay as in <a href="https://discuss.pytorch.org/t/why-batch-loading-time-varies-for-different-batches-image-loading/117107/4">this post</a>.</p>
<p>But it seems that this solution works well only with DataParallel, but not with DistributedDataParallel.</p>
<ul>
<li>
<p>DataParallel num_workers=16<br>
It is working fine because only the first batch takes 3 seconds and all consecutive batches takes almost no time<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/9/4/9411044a2cfcd20e323428a40289766b0cab2d57.png" alt="image" data-base62-sha1="l7R7G7grsZRMWGfW2luUWshMm1N" width="460" height="292"></p>
</li>
<li>
<p>DistributedDataParallel num_workers=16<br>
The first batch is taking 13 seconds to load, which is too much! and as I add more workers it takes even more.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/6/8/68a1a161ee5b0a735e1885b2b1ad11a1d1ddacc4.png" alt="image" data-base62-sha1="eVBWNTfkloU9Pf0koecdMBusrTC" width="457" height="302"></p>
</li>
</ul>
<p>My explanation is that the data sampler is adding more overhead in DistributedDataParallel which is not the case in DataParallel.</p>
<p>Below again is my data sampler code to check if there is any issue with it or potential enhancement:</p>
<pre><code class="lang-auto">batch_size = 100
model = nn.parallel.DistributedDataParallel(model, device_ids=[gpu])
train_sampler = torch.utils.data.distributed.DistributedSampler(training_dataset,
                                                                    num_replicas=args.world_size,
                                                                    rank=rank)
training_dataloader = torch.utils.data.DataLoader(
        dataset=training_dataset,
        batch_size=batch_size,
        shuffle=False,
        num_workers=16,
        sampler=train_sampler,
        pin_memory=True)
</code></pre>
<p>I hope <a class="mention" href="/u/vitalyfedyunin">@VitalyFedyunin</a> and <a class="mention" href="/u/glaringlee">@glaringlee</a> can get back to us for advice about DataLoader and DataSampler.</p>

Jinsuby: <p>Hello, Do you solve these problems? I have the same problems.</p>

H-Huang: <p>Can you open a new issue with the dataloader tag and a script of your issue. This issue is 2 years old now so there have been many changes.</p>