Strange behavior observed for different GPUs

1063082756l: <p>We implemented a DeepLab V3+ integrated with gradient reversal layer (GRL for short) and a domain classifier in order to achieve domain adaptation for some segmentation tasks. The code is at <a href="https://github.com/HaoyuLu1022/DANN-RTS-Segmentation" rel="noopener nofollow ugc">https://github.com/HaoyuLu1022/DANN-RTS-Segmentation</a>.</p>
<p>We initially trained on a single RTX 3090, and it converged as we wished.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/6/6/66785af162af376662335058d7f5f1b2c79f1661.png" alt="epoch_loss" data-base62-sha1="eCuyPZ1CuTYOFqhoTWRrIRdMmR3" width="480" height="360"></p>
<p>Later we began to train the model on a single RTX A6000 with exactly the same environment (say, PyTorch version, cuda toolkit version and even python version) as before. But the model seemed not to converge at all.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/5/b53aea1d45440e9be7ba3d3cf2e739cdc2b84d40.png" data-download-href="https://discuss.pytorch.org/uploads/default/b53aea1d45440e9be7ba3d3cf2e739cdc2b84d40" title="179fbaf19d774a05b992af7f77120b7"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/5/b53aea1d45440e9be7ba3d3cf2e739cdc2b84d40_2_517x357.png" alt="179fbaf19d774a05b992af7f77120b7" data-base62-sha1="pReE2ReHxQyjWl4OUTY99Wuv4s0" width="517" height="357" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/5/b53aea1d45440e9be7ba3d3cf2e739cdc2b84d40_2_517x357.png, https://discuss.pytorch.org/uploads/default/original/3X/b/5/b53aea1d45440e9be7ba3d3cf2e739cdc2b84d40.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/b/5/b53aea1d45440e9be7ba3d3cf2e739cdc2b84d40.png 2x" data-dominant-color="F8F6F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">179fbaf19d774a05b992af7f77120b7</span><span class="informations">751×520 59.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>We did some initial debugging, by first training the original DeepLab V3+ model, and then adding the components (GRL, and then some other tricks for further optimization on performance) and training to see if it converges. And it turned out that the original DeepLab V3+ converged normally, and did not converge right after adding the GRL, which is implemented using <code>autograd.Function</code>. So we thought that it’s probably because of the GRL, but still failed to find a way to solve this.</p>
<p>So could you help us deal with this problem. Much gratitude for your help!</p>
<p>P.S. The dataset is not open-sourced for security reasons, and you may have to test based on other open-source dataset for transfer learning/domain adaptation, which I believe is not the reason for strange convergence in different GPUs. Sorry.</p>

ptrblck: <aside class="quote no-group" data-username="1063082756l" data-post="1" data-topic="187690">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/1063082756l/48/60678_2.png" class="avatar"> 1063082756l:</div>
<blockquote>
<p>So we thought that it’s probably because of the GRL, but still failed to find a way to solve this.</p>
</blockquote>
</aside>
<p>What did you try to isolate the issue so far? It seems GRL refers to a custom <code>autograd.Function</code>, which you have developed and which seems to cause the divergence?</p>

1063082756l: <p>Yes GRL is a custom <code>autograd.Function</code> for reversing the gradients.</p>
<p>And by “isolate”, we only tried removing GRL and the corresponding domain classifier to test whether the original DeepLab V3+ model converges, and it converged normally. We have also tried some open-source implementations of GRL on GitHub, and the GRL-integrated model still didn’t converge. However, we didn’t try something like testing our custom GRL in other models, since our GPU resources are limited.</p>
<p>We suspect that it might be something about compatibility of different hardware architectures, since the same code runs quite differently on 3090 and A6000.</p>

ptrblck: <aside class="quote no-group" data-username="1063082756l" data-post="3" data-topic="187690">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/1063082756l/48/60678_2.png" class="avatar"> 1063082756l:</div>
<blockquote>
<p>We suspect that it might be something about compatibility of different hardware architectures, since the same code runs quite differently on 3090 and A6000.</p>
</blockquote>
</aside>
<p>I don’t know what “compatibility of different hardware architectures” means and to further debug we would need more information and in the ideal case a minimal and executable code snippet showing the difference in convergence.<br>
As a quick test you could disable TF32 via <code>torch.backends.cudnn.allow_tf32 = False</code> (<code>matmuls</code> are not using TF32 by default but to make sure it’s also disabled use <code>torch.backends.cuda.matmul.allow_tf32 = False</code>) and check if this would make a difference.</p>

1063082756l: <p>OK I try that. Thank you.</p>

1063082756l: <p>With <code>torch.backends.cudnn.allow_tf32 = False</code>, training on A6000 still did not converge, and the loss exhibited similar patterns as before.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/2/6/26c70f9eb98bca7e59c129fa575881f2b3dd969d.png" alt="image" data-base62-sha1="5x2D4f79JQGNDf3OdEVECBhMtE9" width="480" height="360"></p>

1063082756l: <p>btw, we freeze the backbone of our model for the first 50 epochs, and then unfreeze it to adjust parameters of the whole model for the rest iterations. So a “jump point” around 50 epochs is also expected just like the original post.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/6/6/66785af162af376662335058d7f5f1b2c79f1661.png" alt="epoch_loss" data-base62-sha1="eCuyPZ1CuTYOFqhoTWRrIRdMmR3" width="480" height="360"></p>