PyTorch Model Reproducibility Issue: RTX 2080 Ti vs. RTX 4090 Ti

MJ-Zeng: <p>Here’s a polished and professional version of your forum post describing the reproducibility issue between RTX 2080 Ti and RTX 4090 Ti:</p>
<hr>
<h3><a name="p-473076-pytorch-model-reproducibility-issue-rtx-2080-ti-vs-rtx-4090-ti-1" class="anchor" href="#p-473076-pytorch-model-reproducibility-issue-rtx-2080-ti-vs-rtx-4090-ti-1"></a>​<strong>PyTorch Model Reproducibility Issue: RTX 2080 Ti vs. RTX 4090 Ti</strong>​</h3>
<p>​<strong>Background:​</strong>​<br>
I’m training the same PyTorch model on two different GPUs but observing significant differences in performance, particularly on the RTX 4090 Ti, which appears to overfit and fails to match the results achieved on the RTX 2080 Ti.</p>
<h4><a name="p-473076-environment-hardware-details-2" class="anchor" href="#p-473076-environment-hardware-details-2"></a>​<strong>Environment &amp; Hardware Details</strong>​</h4>
<p>​<strong>1. RTX 2080 Ti (Original Training Setup)​</strong>​</p>
<pre><code class="lang-auto">{
    'CUDA available': True,
    'CUDA_HOME': None,
    'GCC': 'gcc (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0',
    'GPU 0': 'NVIDIA GeForce RTX 2080 Ti',
    'LibMTL': 'modify 1.1.6',
    'Numpy': '1.21.5',
    'Platform': 'linux',
    'PyTorch': '1.8.0',
    'Python': '3.7.0 (default, Oct 9 2018, 10:31:47) [GCC 7.3.0]',
    'TorchVision': '0.9.0'
}
</code></pre>
<p>​<strong>Epoch 5 Results (2080 Ti):​</strong>​</p>
<pre><code class="lang-auto">[2022-11-29 12:02:56,707][INFO][recorder.py][85] - Epoch: 5 ----- Mode: train
{'buchwald': {'loss': 0.53, 'metrics': {'R2': 0.715}},
 'time': {'backward_time': 12.35, 'data_time': 1.145, 'forward_time': 8.286}}
----------------------------------------------------------------------------------------------------
[2022-11-29 12:02:59,903][INFO][recorder.py][85] - Epoch: 5 ----- Mode: val
{'buchwald': {'loss': 0.537, 'metrics': {'R2': 0.698}},
 'time': {'data_time': 0.014, 'forward_time': 2.049}}
----------------------------------------------------------------------------------------------------
[2022-11-29 12:03:05,232][INFO][recorder.py][85] - Epoch: 5 ----- Mode: test
{'buchwald': {'loss': 0.502, 'metrics': {'R2': 0.749}},
 'time': {'data_time': 0.025, 'forward_time': 4.171}}
</code></pre>
<p>​<strong>2. RTX 4090 Ti (New Training Setup)​</strong>​</p>
<pre><code class="lang-auto">{
    'CUDA available': True,
    'CUDA_HOME': '/usr/local/cuda',
    'GCC': 'gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0',
    'GPU 0,1,2,3,4,5': 'NVIDIA GeForce RTX 4090',
    'LibMTL': 'modify 1.1.6',
    'NVCC': 'Cuda compilation tools, release 11.8, V11.8.89',
    'Numpy': '2.0.2',
    'Platform': 'linux',
    'PyTorch': '2.7.1+cu128',
    'Python': '3.9.23 (main, Jun 5 2025, 13:40:20) [GCC 11.2.0]',
    'TorchVision': '0.22.1+cu128'
}
</code></pre>
<p>​<strong>Epoch 5 Results (4090 Ti):​</strong>​</p>
<pre><code class="lang-auto">[2025-07-17 18:02:06,172][INFO][recorder.py][85] - Epoch: 5 ----- Mode: train
{'buchwald': {'loss': np.float64(0.284), 'metrics': {'R2': 0.924}},
 'time': {'backward_time': 5.906, 'data_time': 0.033, 'forward_time': 10.398}}
----------------------------------------------------------------------------------------------------
[2025-07-17 18:02:08,721][INFO][recorder.py][85] - Epoch: 5 ----- Mode: val
{'buchwald': {'loss': np.float64(0.631), 'metrics': {'R2': 0.593}},
 'time': {'data_time': 0.001, 'forward_time': 2.539}}
----------------------------------------------------------------------------------------------------
[2025-07-17 18:02:14,147][INFO][recorder.py][85] - Epoch: 5 ----- Mode: test
{'buchwald': {'loss': np.float64(0.549), 'metrics': {'R2': 0.71}},
 'time': {'data_time': 0.0, 'forward_time': 5.41}}
</code></pre>
<h4><a name="p-473076-key-observations-3" class="anchor" href="#p-473076-key-observations-3"></a>​<strong>Key Observations:​</strong>​</h4>
<ol>
<li>​<strong>Overfitting on 4090 Ti</strong>:</li>
</ol>
<ul>
<li>The training loss (<code>0.284</code>) is much lower than on the 2080 Ti (<code>0.53</code>), but the validation (<code>0.631</code> vs. <code>0.537</code>) and test (<code>0.549</code> vs. <code>0.502</code>) losses are worse.</li>
<li>The <code>R2</code> metric drops significantly on validation (<code>0.593</code> vs. <code>0.698</code>) and test (<code>0.71</code> vs. <code>0.749</code>).</li>
</ul>
<ol start="2">
<li>​<strong>Failed to Reproduce Original Performance</strong>:</li>
</ol>
<ul>
<li>Even after hyperparameter tuning, the model on the 4090 Ti cannot reach the previous best <code>R2</code> of <code>0.95</code> achieved on the 2080 Ti.</li>
</ul>
<h4><a name="p-473076-possible-causes-questions-4" class="anchor" href="#p-473076-possible-causes-questions-4"></a>​<strong>Possible Causes &amp; Questions:​</strong>​</h4>
<ul>
<li>​<strong>Numerical Differences</strong>: Are there any known non-determinism issues between PyTorch 1.8.0 (2080 Ti) and 2.7.1 (4090 Ti)?</li>
<li>​<strong>Hardware Differences</strong>: Does the 4090 Ti’s architecture (Ada Lovelace vs. Turing) affect training dynamics?</li>
</ul>
<p>​<strong>Any insights or suggestions would be greatly appreciated!"​</strong>​</p>
<hr>
<p>Would you like me to suggest potential solutions (e.g., deterministic training flags, mixed precision adjustments)?</p>

ptrblck: <p>The differences between the used setups are quite large as PyTorch 1.8.0 was released in March 2021 while 2.7.1 was released in May 2025. It could help if you could double check the outputs using any versions between these releases and also it could help to measure the reproducibility of these different outputs making sure something indeed changed and you are not measuring random noise.</p>