Speed drop with dilated conv on different GPUs

username: tree33
cooked: <p>Hi, I’m experiencing a huge speed drop when running the exact same code on pascal cards(like 1080, or titanXP, or P100) and more latest cards(titanV or 2080tis), both running on pytorch1.5, cuda10.2 and cudnn7.6.5 shipped with pytorch.</p>
<p>After looking into the code, I found out the problem is about the dilated conv in my network.</p>
<p>I’m running a task similar to image segmentation and I’m using a resnet2d as my network. Here for comparison, I turned off the shuffle on my dataloaders, so input size should be the exact same in all my benchmarks.<br>
When I set all dilations to 1 in all my network, the speed is quite similar on different cards, around 30~35 seconds for 100 iterations on both pascals cards and latest cards.</p>
<p>However, when I turn dilation on, speed would drop significantly on pascal cards, around 140 seconds for 100 iterations, while on titanV, it’s still around 40 seconds for 100 iterations.</p>
<p>In short, speed is similar when I turn dilation off, and speed on pascal cards are much slower (3 to 4 times )when I turn dilation on.</p>
<p>I found a similar topic here, discussing about the backend cudnn here.<br>
</p><aside class="onebox githubissue">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/issues/15054" target="_blank" rel="nofollow noopener">github.com/pytorch/pytorch</a>
  </header>
  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewBox="0 0 14 16" aria-hidden="true"><path d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/issues/15054" target="_blank" rel="nofollow noopener">Dilated conv in v1.0.0 is too slow.</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2018-12-11" data-time="11:13:32" data-timezone="UTC">11:13AM - 11 Dec 18 UTC</span>
      </div>

        <div class="date">
          closed <span class="discourse-local-date" data-format="ll" data-date="2018-12-11" data-time="11:41:04" data-timezone="UTC">11:41AM - 11 Dec 18 UTC</span>
        </div>

      <div class="user">
        <a href="https://github.com/qinjian623" target="_blank" rel="nofollow noopener">
          <img alt="qinjian623" src="https://avatars1.githubusercontent.com/u/1606866?v=4" class="onebox-avatar-inline" width="20" height="20">
          qinjian623
        </a>
      </div>
    </div>
  </div>
</div>

<div class="github-row">
  <p class="github-content">🐛 Bug
Hi, i've ported my model to 1.0.0, then found out my model much slower then it on v0.4.1.
So I did...</p>
</div>

<div class="labels">
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>However, as my input size would change,  setting torch.backends.cudnn.benchmark to True just slow my speed down. Any help here?</p>

username: ptrblck
cooked: <p>Could you disable cudnn via <code>torch.backends.cudnn.enabled = False</code> and rerun the check?<br>
If the performance numbers are closer to each other, then most likely cudnn’s default algorithm for the Pascal architecture is slower than for Volta.<br>
How different are your input shapes? Are you getting new shapes in each iteration or only in the first couple of iterations?</p>

username: tree33
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="82412">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>How different are your input shapes? Are you getting new shapes in each iteration or only in the first couple of iterations?</p>
</blockquote>
</aside>
<p>Hi, thanks for your help! I disabled cudnn by setting cudnn.enable to False in my experiments.<br>
I also used a shallow resnet here, because disabling the flag seems to raise the memory usage.<br>
Now the speed are quite in similar in 1080tis and RTX2080tis, both around 24s~30s for 100 iterations.</p>
<p>My task is similar to image segmentation, the input here is a Lx20 (20 is the feature) tensor and outputs a L*L 0/1 segmentation map. L ranges from 50 to up to 400. Batch size is set to 1,  and I’m getting new shapes in every iteration. ehhh I don’t think resizing the input to a fixed L looks reasonable in my task.</p>