Potential gradient precision problem for RTX 2080Ti?

username: teracamo
cooked: <p>I am experiencing a strange loss behavior when I migrated my coded to an RTX2080Ti machine from a Titan XP machine.</p>
<p>I am getting a large loss fluctuation when the loss is becoming small. I have suspicions that its a precision problem but I didn’t get any under flow error during training.</p>
<p>I have read some post about factory bad GPUs but I have tried identical code on 4 separate piece of GPU, they all have similar problems.</p>
<p>The loss is like this:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/0/0fb340696f58f0db475cef6c5f917ac7006f429c.png" data-download-href="https://discuss.pytorch.org/uploads/default/0fb340696f58f0db475cef6c5f917ac7006f429c" title="%E5%9C%96%E7%89%87"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/0/0fb340696f58f0db475cef6c5f917ac7006f429c_2_690x180.png" alt="%E5%9C%96%E7%89%87" data-base62-sha1="2eTcP9UFl9Sqi5NI9R3vwiI65vS" width="690" height="180" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/0/0fb340696f58f0db475cef6c5f917ac7006f429c_2_690x180.png, https://discuss.pytorch.org/uploads/default/optimized/2X/0/0fb340696f58f0db475cef6c5f917ac7006f429c_2_1035x270.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/0/0fb340696f58f0db475cef6c5f917ac7006f429c_2_1380x360.png 2x" data-dominant-color="F0F0F1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">%E5%9C%96%E7%89%87</span><span class="informations">1446×378 142 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Four colors represents different GPUs on RTX2080Ti</p>
<p>However, the validation loss looks fine , can’t upload that due to new users <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=12" title=":frowning:" class="emoji" alt=":frowning:" loading="lazy" width="20" height="20"><br>
I am using PyTorch 1.2, Cuda 10.0 from Anaconda3 on an Ubuntu 18.0 machine.<br>
Any comments are welcomed cause this had bothered me for quite some time.</p>

username: teracamo
cooked: <p>I was using SGD with LR: 1E-4 Momentum 0.5</p>

username: ptrblck
cooked: <p>Could you post the loss on your Titan XP?<br>
Are there any other differences regarding the CUDA, cudnn, PyTorch versions between your Titan XP and RTX2080Ti runs?</p>

username: teracamo
cooked: <p>Its something like this.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/b/b2049a0a44dd9780090a2779bd4a32c944f257d0.png" data-download-href="https://discuss.pytorch.org/uploads/default/b2049a0a44dd9780090a2779bd4a32c944f257d0" title="%E5%9C%96%E7%89%87"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/b/b2049a0a44dd9780090a2779bd4a32c944f257d0_2_690x180.png" alt="%E5%9C%96%E7%89%87" data-base62-sha1="poOQD5m3YuzOxOhNfw4MzIgceUE" width="690" height="180" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/b/b2049a0a44dd9780090a2779bd4a32c944f257d0_2_690x180.png, https://discuss.pytorch.org/uploads/default/optimized/2X/b/b2049a0a44dd9780090a2779bd4a32c944f257d0_2_1035x270.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/b/b2049a0a44dd9780090a2779bd4a32c944f257d0_2_1380x360.png 2x" data-dominant-color="FAFBFC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">%E5%9C%96%E7%89%87</span><span class="informations">1446×378 38.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
As seen its much smoother at small losses and converge much better.<br>
I only have one Titan Xp GPU though.</p>
<p>I used CUDA 9.0  with cudnn 7501 and Pytorch 1.1.</p>
<p>RTX 2080Ti used cudnn 7602 acording to torch.backends.cudnn.version()</p>
<p>Do you think its a version problem? I wouldn’t be able to downgrade to CUDA 9.2 for RTX2080Ti I suppose.</p>

username: ptrblck
cooked: <p>Yeah, you shouldn’t use older CUDA versions than 10.0 for your RTX2080Ti, but could you rerun the script on your Titan XP using CUDA10.0, cudnn 7.6.2 and PyTorch 1.1?<br>
It would give us an idea, where the difference might come from.</p>

username: teracamo
cooked: <p>I haven’t been able to do as many steps as the blue curve, but here’s the result:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/3/324f2aa523acc2254bf0310b02257c6973bc8351.png" data-download-href="https://discuss.pytorch.org/uploads/default/324f2aa523acc2254bf0310b02257c6973bc8351" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/3/324f2aa523acc2254bf0310b02257c6973bc8351_2_689x210.png" alt="image" data-base62-sha1="7b3uczMfvCBh4SpK8UKaM5rAEUN" width="689" height="210" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/3/324f2aa523acc2254bf0310b02257c6973bc8351_2_689x210.png, https://discuss.pytorch.org/uploads/default/optimized/2X/3/324f2aa523acc2254bf0310b02257c6973bc8351_2_1033x315.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/3/324f2aa523acc2254bf0310b02257c6973bc8351.png 2x" data-dominant-color="FCF9F8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1240×378 30.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Seems a lot less jumpy than 2080 already when the loss gets small though more rough than the blue curve.</p>
<p>I think that tells there’s something wrong with 2080Ti?</p>

username: teracamo
cooked: <p>I have also noticed that one of GPU is significantly hotter than the others. Don’t know if its related.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/2X/c/c7112d7efaea7abd70fd40d0ab49f28934e30739.png" alt="image" data-base62-sha1="sp1OOYmFcw2ifN3mzAzbZAYVMSJ" width="410" height="58"></p>

username: ptrblck
cooked: <p>I’m not sure at this point, how to interpret the loss curves.<br>
The blue (good) curve looks like the x-axis is sampled differently.<br>
Especially at the beginning it looks like the step size is not constant and you have some interpolated lines.<br>
Are you using the same step size (e.g. wall time, iterations)?</p>
<p>Also, does the y-axis have the same range in all plots? If the blue line had some really high loss at the beginning, the rest might look smoother as it’s just smaller.</p>

username: teracamo
cooked: <p>The blue one does have a lot more steps, sorry for not including the steps count</p>
<p>But I think its also clear that the RTX2080Ti run with a lot more fluctuations than Titan Xp.<br>
I one of the run from RTX2080Ti and removed the smoothing for comparison.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/d/d1609a467aede8512d4b9967e9732464f85c6d12.png" data-download-href="https://discuss.pytorch.org/uploads/default/d1609a467aede8512d4b9967e9732464f85c6d12" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/d/d1609a467aede8512d4b9967e9732464f85c6d12_2_690x195.png" alt="image" data-base62-sha1="tSeLdhNKFVjHBX1yP9Mgl9VwyEG" width="690" height="195" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/d/d1609a467aede8512d4b9967e9732464f85c6d12_2_690x195.png, https://discuss.pytorch.org/uploads/default/optimized/2X/d/d1609a467aede8512d4b9967e9732464f85c6d12_2_1035x292.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/d/d1609a467aede8512d4b9967e9732464f85c6d12_2_1380x390.png 2x" data-dominant-color="FDF3F0"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1418×401 34.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: teracamo
cooked: <p>And this is the Titan Xp results without smoothing, identical versions and data.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/1/139439b87484bbc6e2d28fd7280857bf591ccc9f.png" data-download-href="https://discuss.pytorch.org/uploads/default/139439b87484bbc6e2d28fd7280857bf591ccc9f" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/1/139439b87484bbc6e2d28fd7280857bf591ccc9f_2_690x197.png" alt="image" data-base62-sha1="2NcDTMOXKZxHUN4380cBn3QqQPB" width="690" height="197" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/1/139439b87484bbc6e2d28fd7280857bf591ccc9f_2_690x197.png, https://discuss.pytorch.org/uploads/default/optimized/2X/1/139439b87484bbc6e2d28fd7280857bf591ccc9f_2_1035x295.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/1/139439b87484bbc6e2d28fd7280857bf591ccc9f_2_1380x394.png 2x" data-dominant-color="FAF3F1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1423×408 39.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>