Apex and Random Seed

username: Hannibal046
cooked: <p>Hi, I am wondering have you ever encountered such situation  when using Apex Fp16, random seed seems to be out of work.<br>
This is how I set random seed</p>
<pre data-code-wrap="python"><code class="lang-python">def setup_seed(seed):
    torch.manual_seed(seed)
    torch.cuda.manual_seed_all(seed)
    np.random.seed(seed)
    random.seed(seed)
    torch.backends.cudnn.deterministic = True
</code></pre>
<p>This is where things get weird. I ran two process simultaneously on GPU0 and GPU1 with same conda environment.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/6/e63eb744ffb21af809ebf9ffb1812a46e4790b0f.png" data-download-href="https://discuss.pytorch.org/uploads/default/e63eb744ffb21af809ebf9ffb1812a46e4790b0f" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/6/e63eb744ffb21af809ebf9ffb1812a46e4790b0f_2_690x178.png" alt="image" data-base62-sha1="wQQbyura3g4mYr3OR67XpRZHE7B" width="690" height="178" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/6/e63eb744ffb21af809ebf9ffb1812a46e4790b0f_2_690x178.png, https://discuss.pytorch.org/uploads/default/optimized/3X/e/6/e63eb744ffb21af809ebf9ffb1812a46e4790b0f_2_1035x267.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/e/6/e63eb744ffb21af809ebf9ffb1812a46e4790b0f_2_1380x356.png 2x" data-dominant-color="1E1A1A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">3002×778 336 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/b/5b1a5a8600b78879786420936e117e7645294e05.png" data-download-href="https://discuss.pytorch.org/uploads/default/5b1a5a8600b78879786420936e117e7645294e05" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/b/5b1a5a8600b78879786420936e117e7645294e05_2_690x129.png" alt="image" data-base62-sha1="cZVUyjOZ3Kma5chTmze4xgSnHil" width="690" height="129" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/b/5b1a5a8600b78879786420936e117e7645294e05_2_690x129.png, https://discuss.pytorch.org/uploads/default/optimized/3X/5/b/5b1a5a8600b78879786420936e117e7645294e05_2_1035x193.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/5/b/5b1a5a8600b78879786420936e117e7645294e05_2_1380x258.png 2x" data-dominant-color="191616"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">2980×558 149 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p><code>apex.amp</code> is deprecated so please use the native mixed-precision implementation via <code>torch.cuda.amp</code> as described <a href="https://pytorch.org/docs/stable/notes/amp_examples.html#amp-examples">here</a>.</p>