Distributed evaluation/inference with different performance per GPU

username: thorstenwagner
cooked: <p>Hi,</p>
<p>I’m implementing distributed evaluation using DistributedDataParallel and so far, with the help of this forum, it works quite well <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>However, I noticed that when running it on two GPUs (Titan V) the second gpu is quite a bit slower than the first one.</p>
<p>The first GPU does around 4 it/s whereas the second GPU only does 3 it/s . I was wondering if there is a problem with my implementation, so here is the the code:</p>
<aside class="onebox githubgist" data-onebox-src="https://gist.github.com/thorstenwagner/744883b406c6c7b481e5876bf44124ee">
  <header class="source">

      <a href="https://gist.github.com/thorstenwagner/744883b406c6c7b481e5876bf44124ee" target="_blank" rel="noopener nofollow ugc">gist.github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://gist.github.com/thorstenwagner/744883b406c6c7b481e5876bf44124ee" target="_blank" rel="noopener nofollow ugc">https://gist.github.com/thorstenwagner/744883b406c6c7b481e5876bf44124ee</a></h4>

  <h5>embed.py</h5>
  <pre><code class="Python">    def embed(self, volume_data: VolumeDataset) -&gt; np.array:
        """Calculates the embeddings. The volumes showed have the dimension NxBSxBSxBS"""

        torch.backends.cudnn.benchmark = True

        dataset = TorchVolumeDataset(volumes=volume_data)
        sampler_data = torch.utils.data.DistributedSampler(dataset, rank=self.rank)
        volume_loader = DataLoader(
            dataset=dataset,
            batch_size=self.batchsize,</code></pre>
   This file has been truncated. <a href="https://gist.github.com/thorstenwagner/744883b406c6c7b481e5876bf44124ee" target="_blank" rel="noopener nofollow ugc">show original</a>

<p>
</p>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Do you see any obvious flaws?</p>
<p>Both GPUs are using roughly the same amount of memory:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/d/a/dab536760d8195b3756ee745e5068b5c7758e9c6.png" alt="image" data-base62-sha1="vcMkLYfpB03mb6NFFPF3noePice" width="640" height="352"></p>
<p>I also noticed that the GPU 1 most of the time uses less power then GPU 0.</p>
<p>Any explanation for performance loss on the second GPU?</p>
<p>Best,<br>
Thorsten</p>
<p>PS:<br>
I’m using pytorch 2.1.0.dev20230719 because of the problems written <a href="https://discuss.pytorch.org/t/combining-torch-compile-and-distributeddataparallel/184591/3">here</a>.</p>

username: ptrblck
cooked: <p>I don’t know what kind of setup you are using, but did you profile the devices in other applications (i.e. not in PyTorch)? E.g. is the PCIe bandwidth the same for all devices or is GPU1 using less lanes etc.?</p>

username: thorstenwagner
cooked: <p>Ok, I found the problem… its simply heat. Opening the computer makes both GPUs running at same speed. Sorry for bothering you with that.</p>

username: ptrblck
cooked: <p>Not at all and thanks for sharing the root cause as it can certainly help others!</p>